[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:36:34 dist-test-slave-lhhq krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:36:34 dist-test-slave-lhhq krb5kdc[395](info): set up 2 sockets
Sep 26 21:36:34 dist-test-slave-lhhq krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 21:36:35.397368 388 mini_kdc.cc:121] Time spent starting KDC: real 2.039s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 21:36:35.528606 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.131s user 0.002s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:36:35 dist-test-slave-lhhq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:36:35.677686 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42915
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:36.012108 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:36.012414 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:36.012599 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:36.019869 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:36:36.020015 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:36.020107 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:36.020215 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:36:36.020300 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:36:36.030491 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:36.031857 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:36.033331 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:36.046712 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:36.046648 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:36.046682 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:36.047513 408 server_base.cc:1034] running on GCE node
I20260926 21:36:36.048303 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:36.049809 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:36.051062 408 hybrid_clock.cc:648] HybridClock initialized: now 1790458596050965 us; error 85 us; skew 500 ppm
I20260926 21:36:36.051404 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:36 dist-test-slave-lhhq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:36.054494 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:36:36.056034 408 webserver.cc:460] Webserver started at http://127.0.97.62:45109/ using document root <none> and password file <none>
I20260926 21:36:36.056733 408 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:36.056870 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:36.057241 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:36.060210 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "8ac7334deb444d389aedd5fdeb48e611"
format_stamp: "Formatted at 2026-09-26 21:36:36 on dist-test-slave-lhhq"
server_key: "dd93975c40cb7d4d761e29034f8f165b"
server_key_iv: "a19e3399b69bd25144bef723644a70fd"
server_key_version: "encryptionkey@0"
I20260926 21:36:36.060968 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ac7334deb444d389aedd5fdeb48e611"
format_stamp: "Formatted at 2026-09-26 21:36:36 on dist-test-slave-lhhq"
server_key: "dd93975c40cb7d4d761e29034f8f165b"
server_key_iv: "a19e3399b69bd25144bef723644a70fd"
server_key_version: "encryptionkey@0"
I20260926 21:36:36.066452 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 21:36:36.070451 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:36.071777 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:36:36.071969 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "8ac7334deb444d389aedd5fdeb48e611"
format_stamp: "Formatted at 2026-09-26 21:36:36 on dist-test-slave-lhhq"
server_key: "dd93975c40cb7d4d761e29034f8f165b"
server_key_iv: "a19e3399b69bd25144bef723644a70fd"
server_key_version: "encryptionkey@0"
I20260926 21:36:36.072180 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:36.089078 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:36.093748 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:36.094074 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:36.114651 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42915
I20260926 21:36:36.114668 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42915 every 8 connection(s)
I20260926 21:36:36.116109 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:36:36.119788 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:36.124612 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 408
I20260926 21:36:36.124897 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:36:36.127786 388 external_mini_cluster.cc:1428] Setting key f7b9bd766ae157675c34032965a53c71
I20260926 21:36:36.132673 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611: Bootstrap starting.
I20260926 21:36:36.135761 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:36.136742 475 log.cc:824] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:36.140036 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611: No bootstrap required, opened a new log
I20260926 21:36:36.150157 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac7334deb444d389aedd5fdeb48e611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42915 } }
I20260926 21:36:36.150628 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:36.150729 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac7334deb444d389aedd5fdeb48e611, State: Initialized, Role: FOLLOWER
I20260926 21:36:36.151367 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac7334deb444d389aedd5fdeb48e611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42915 } }
I20260926 21:36:36.151608 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:36:36.151767 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:36:36.151923 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:36.154722 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac7334deb444d389aedd5fdeb48e611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42915 } }
I20260926 21:36:36.155203 475 leader_election.cc:302] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ac7334deb444d389aedd5fdeb48e611; no voters:
I20260926 21:36:36.155956 475 leader_election.cc:288] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:36:36.156121 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:36.157377 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [term 1 LEADER]: Becoming Leader. State: Replica: 8ac7334deb444d389aedd5fdeb48e611, State: Running, Role: LEADER
I20260926 21:36:36.157953 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac7334deb444d389aedd5fdeb48e611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42915 } }
I20260926 21:36:36.158372 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 21:36:36 dist-test-slave-lhhq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:36.161731 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ac7334deb444d389aedd5fdeb48e611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac7334deb444d389aedd5fdeb48e611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42915 } } }
I20260926 21:36:36.162122 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:36.163517 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ac7334deb444d389aedd5fdeb48e611. Latest consensus state: current_term: 1 leader_uuid: "8ac7334deb444d389aedd5fdeb48e611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac7334deb444d389aedd5fdeb48e611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42915 } } }
I20260926 21:36:36.163887 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611 [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:36.165825 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:36:36.171876 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:36:36.176651 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:36.132446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.0.97.62:42915)
0926 21:36:36.133356 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:36.133371 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:36.134639 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 21:36:36.144918 (+ 10279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:36.144945 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:36.145498 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:36.145962 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:36.148684 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:36.151142 (+ 2458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:36.152969 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:36.154042 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:36.154232 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:36.161531 (+ 7299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:36.161547 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:36.161572 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:36.161649 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:36.166655 (+ 5006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:36.167897 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:36.167912 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:36.167925 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:36.168225 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:36.169012 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:36.169024 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:36.169029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:36.170033 (+ 1004us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:36.173843 (+ 3810us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:36.174205 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 21:36:36.174811 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":364,"threads_started":1}
I20260926 21:36:36.189065 486 catalog_manager.cc:1347] Generated new cluster ID: e046bac486b942d5b5e0014a8eef760c
I20260926 21:36:36.189217 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:36:36.206961 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:36:36.208279 486 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:36:36.219532 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ac7334deb444d389aedd5fdeb48e611: Generated new TSK 0
I20260926 21:36:36.220566 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:36:36.273142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42915
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:36.625454 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:36.625748 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:36.625918 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:36.633028 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:36.633145 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:36.633384 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:36:36.643301 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:36.644676 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:36.646060 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:36.659854 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:36.659799 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:36.660866 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:36.661180 501 server_base.cc:1034] running on GCE node
I20260926 21:36:36.661902 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:36.662863 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:36.664070 501 hybrid_clock.cc:648] HybridClock initialized: now 1790458596664025 us; error 36 us; skew 500 ppm
I20260926 21:36:36.664427 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:36 dist-test-slave-lhhq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:36.667724 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:36:36.669271 501 webserver.cc:460] Webserver started at http://127.0.97.1:37983/ using document root <none> and password file <none>
I20260926 21:36:36.669989 501 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:36.670125 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:36.670450 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:36.673576 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "deb94772dd354faf91ec27b9055cb4fc"
format_stamp: "Formatted at 2026-09-26 21:36:36 on dist-test-slave-lhhq"
server_key: "c8d2ce4e0dd8ab74d4726a6a7e01e7c3"
server_key_iv: "6cf7499f73ba41da4906971edb5c8146"
server_key_version: "encryptionkey@0"
I20260926 21:36:36.674736 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "deb94772dd354faf91ec27b9055cb4fc"
format_stamp: "Formatted at 2026-09-26 21:36:36 on dist-test-slave-lhhq"
server_key: "c8d2ce4e0dd8ab74d4726a6a7e01e7c3"
server_key_iv: "6cf7499f73ba41da4906971edb5c8146"
server_key_version: "encryptionkey@0"
I20260926 21:36:36.680070 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:36:36.683607 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:36.685170 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:36:36.685393 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "deb94772dd354faf91ec27b9055cb4fc"
format_stamp: "Formatted at 2026-09-26 21:36:36 on dist-test-slave-lhhq"
server_key: "c8d2ce4e0dd8ab74d4726a6a7e01e7c3"
server_key_iv: "6cf7499f73ba41da4906971edb5c8146"
server_key_version: "encryptionkey@0"
I20260926 21:36:36.685575 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:36.736222 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:36.740794 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:36.741042 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:36.741974 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:36:36.743245 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:36.743315 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:36.743399 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:36.743443 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:36.779516 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40741
I20260926 21:36:36.779620 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40741 every 8 connection(s)
I20260926 21:36:36.780892 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:36:36.781822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 501
I20260926 21:36:36.782024 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:36:36.782815 388 external_mini_cluster.cc:1428] Setting key e2f8e46427f2815efe584040542bcde9
Sep 26 21:36:36 dist-test-slave-lhhq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:36:36.805605 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:36.784140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55029 (local address 127.0.97.62:42915)
0926 21:36:36.784625 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:36.784631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:36.786282 (+ 1651us) server_negotiation.cc:407] Connection header received
0926 21:36:36.789852 (+ 3570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:36.789861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:36.789954 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:36.790068 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:36.791702 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:36.792972 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:36.794649 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:36.795420 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:36.795578 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:36.799550 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:36.799556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:36.799559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:36.799593 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:36.802271 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:36.803109 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:36.803116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:36.803141 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:36.803288 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:36.803760 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:36.803765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:36.803769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:36.804070 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:36.804231 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:36.804965 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 21:36:36.805239 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":257,"threads_started":1}
I20260926 21:36:36.806641 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:36.784795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42915 (local address 127.0.97.1:55029)
0926 21:36:36.786097 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:36.786153 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:36.789394 (+ 3241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:36.790358 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:36.790368 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:36.790916 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:36.791506 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:36.791525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:36.793142 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:36.793146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:36.794485 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:36.794497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:36.795647 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:36.795652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:36.795862 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:36.796639 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:36.796674 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:36.799283 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:36.802433 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:36.802444 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:36.802469 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:36.802910 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:36.803404 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:36.803410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:36.803414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:36.803632 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:36.804202 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:36.804210 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:36.804579 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 21:36:36.804868 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 21:36:36.805464 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":807,"threads_started":1}
I20260926 21:36:36.808429 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42915
I20260926 21:36:36.808827 629 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:36.809859 629 heartbeater.cc:511] Master 127.0.97.62:42915 requested a full tablet report, sending...
I20260926 21:36:36.812361 440 ts_manager.cc:194] Registered new tserver with Master: deb94772dd354faf91ec27b9055cb4fc (127.0.97.1:40741)
I20260926 21:36:36.813977 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:36:36.828276 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42915
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:37.172108 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:37.172399 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:37.172580 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:37.180109 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:37.180229 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:37.180408 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:36:37.191143 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:37.192451 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:37.193931 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:37.207947 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:37.207952 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:37.207957 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:37.208916 635 server_base.cc:1034] running on GCE node
I20260926 21:36:37.209653 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:37.210721 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:37.211925 635 hybrid_clock.cc:648] HybridClock initialized: now 1790458597211858 us; error 67 us; skew 500 ppm
I20260926 21:36:37.212276 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:37 dist-test-slave-lhhq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:37.219733 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:36:37.221522 635 webserver.cc:460] Webserver started at http://127.0.97.2:45919/ using document root <none> and password file <none>
I20260926 21:36:37.222276 635 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:37.222429 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:37.222759 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:37.225886 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "c38f4852f1184759b69e8eeaa70d3e3d"
format_stamp: "Formatted at 2026-09-26 21:36:37 on dist-test-slave-lhhq"
server_key: "23e25c33c9b89645f25f5438b0587a2f"
server_key_iv: "0003407a1f4efd2e9eb82c7ac1961fa7"
server_key_version: "encryptionkey@0"
I20260926 21:36:37.226727 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c38f4852f1184759b69e8eeaa70d3e3d"
format_stamp: "Formatted at 2026-09-26 21:36:37 on dist-test-slave-lhhq"
server_key: "23e25c33c9b89645f25f5438b0587a2f"
server_key_iv: "0003407a1f4efd2e9eb82c7ac1961fa7"
server_key_version: "encryptionkey@0"
I20260926 21:36:37.232276 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:36:37.236333 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:37.238040 635 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 21:36:37.238257 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "c38f4852f1184759b69e8eeaa70d3e3d"
format_stamp: "Formatted at 2026-09-26 21:36:37 on dist-test-slave-lhhq"
server_key: "23e25c33c9b89645f25f5438b0587a2f"
server_key_iv: "0003407a1f4efd2e9eb82c7ac1961fa7"
server_key_version: "encryptionkey@0"
I20260926 21:36:37.238493 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:37.255568 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:37.260282 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:37.260586 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:37.261654 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:36:37.263053 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:37.263135 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:37.263299 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:37.263363 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:37.299515 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33951
I20260926 21:36:37.299556 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33951 every 8 connection(s)
I20260926 21:36:37.300890 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:36:37.305753 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 635
I20260926 21:36:37.305918 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:36:37.306463 388 external_mini_cluster.cc:1428] Setting key 09c87619e392bc6fd8757e129a725005
Sep 26 21:36:37 dist-test-slave-lhhq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:37.324347 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:37.303430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53171 (local address 127.0.97.62:42915)
0926 21:36:37.303612 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:37.303617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:37.304780 (+ 1163us) server_negotiation.cc:407] Connection header received
0926 21:36:37.307220 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:37.307229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:37.307327 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:37.307408 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:37.309347 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.310010 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.313015 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.313543 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.313708 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:37.317767 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:37.317772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:37.317775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:37.317809 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:37.320875 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.321911 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.321915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.321917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.322075 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.322605 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.322608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.322610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.322862 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:37.322971 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:37.323914 (+ 943us) server_negotiation.cc:299] Negotiation successful
0926 21:36:37.324108 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:36:37.325804 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:37.303932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42915 (local address 127.0.97.2:53171)
0926 21:36:37.304617 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:37.304658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:37.306880 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:37.307710 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:37.307722 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:37.308379 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:37.309162 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:37.309210 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.310175 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.310183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:37.312830 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:37.312848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.313803 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.313810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:37.314060 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:37.314817 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:37.314851 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:37.317532 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:37.321038 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:37.321050 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:37.321071 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:37.321768 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:37.322198 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:37.322206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:37.322212 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:37.322476 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:37.322993 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:37.323003 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:37.323491 (+ 488us) client_negotiation.cc:769] Sending connection context
0926 21:36:37.323819 (+ 328us) client_negotiation.cc:240] Negotiation successful
0926 21:36:37.324408 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":275,"threads_started":1}
I20260926 21:36:37.327209 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42915
I20260926 21:36:37.327476 763 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 21:36:37.328246 763 heartbeater.cc:511] Master 127.0.97.62:42915 requested a full tablet report, sending...
I20260926 21:36:37.329975 440 ts_manager.cc:194] Registered new tserver with Master: c38f4852f1184759b69e8eeaa70d3e3d (127.0.97.2:33951)
I20260926 21:36:37.331250 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:36:37.352703 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42915
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:37.709764 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:37.710106 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:37.710283 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:37.717461 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:37.717635 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:37.717832 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:36:37.731815 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:37.733242 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:37.734659 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:37.748219 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:37.748281 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:37.748299 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:37.749162 768 server_base.cc:1034] running on GCE node
I20260926 21:36:37.749806 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:37.750767 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:37.752007 768 hybrid_clock.cc:648] HybridClock initialized: now 1790458597751936 us; error 44 us; skew 500 ppm
I20260926 21:36:37.752370 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:37 dist-test-slave-lhhq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:37.755541 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:36:37.756975 768 webserver.cc:460] Webserver started at http://127.0.97.3:44379/ using document root <none> and password file <none>
I20260926 21:36:37.757723 768 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:37.757833 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:37.758118 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:37.761709 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "17fa0db9e50242d191a9d0a9ce20aec1"
format_stamp: "Formatted at 2026-09-26 21:36:37 on dist-test-slave-lhhq"
server_key: "4d79b4954df689fc078a9f033c464fdd"
server_key_iv: "f95ac4bfa66cb14751b6902081dd2cf3"
server_key_version: "encryptionkey@0"
I20260926 21:36:37.762459 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "17fa0db9e50242d191a9d0a9ce20aec1"
format_stamp: "Formatted at 2026-09-26 21:36:37 on dist-test-slave-lhhq"
server_key: "4d79b4954df689fc078a9f033c464fdd"
server_key_iv: "f95ac4bfa66cb14751b6902081dd2cf3"
server_key_version: "encryptionkey@0"
I20260926 21:36:37.768110 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 21:36:37.771970 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:37.773687 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:36:37.773841 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "17fa0db9e50242d191a9d0a9ce20aec1"
format_stamp: "Formatted at 2026-09-26 21:36:37 on dist-test-slave-lhhq"
server_key: "4d79b4954df689fc078a9f033c464fdd"
server_key_iv: "f95ac4bfa66cb14751b6902081dd2cf3"
server_key_version: "encryptionkey@0"
I20260926 21:36:37.774027 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:37.789840 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:37.794111 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:37.794361 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:37.795324 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:36:37.796669 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:37.796747 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:37.796830 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:37.796876 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:37.818285 629 heartbeater.cc:503] Master 127.0.97.62:42915 was elected leader, sending a full tablet report...
I20260926 21:36:37.833858 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34161
I20260926 21:36:37.833874 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34161 every 8 connection(s)
I20260926 21:36:37.835201 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:36:37.844241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 768
I20260926 21:36:37.844466 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:36:37.845049 388 external_mini_cluster.cc:1428] Setting key 67539ebf67dca3d62da0b529166c65f7
Sep 26 21:36:37 dist-test-slave-lhhq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:37.857321 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:37.837824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38439 (local address 127.0.97.62:42915)
0926 21:36:37.838257 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:37.838264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:37.839796 (+ 1532us) server_negotiation.cc:407] Connection header received
0926 21:36:37.841656 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:37.841660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:37.841729 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:37.841877 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:37.843440 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.844354 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.846659 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.847097 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.847249 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:37.851397 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:37.851403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:37.851406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:37.851442 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:37.854046 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.854858 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.854863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.854865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.854986 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.855408 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.855411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.855413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.855682 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:37.855819 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:37.856732 (+ 913us) server_negotiation.cc:299] Negotiation successful
0926 21:36:37.856920 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":196,"threads_started":1}
I20260926 21:36:37.858143 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:37.838463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42915 (local address 127.0.97.3:38439)
0926 21:36:37.839621 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:37.839664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:37.841370 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:37.842074 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:37.842082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:37.842642 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:37.843283 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:37.843300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.844554 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.844559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:37.846519 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:37.846534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.847337 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.847342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:37.847664 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:37.848487 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:37.848510 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:37.851134 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:37.854219 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:37.854227 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:37.854240 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:37.854701 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:37.855100 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:37.855104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:37.855106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:37.855299 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:37.855830 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:37.855842 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:37.856283 (+ 441us) client_negotiation.cc:769] Sending connection context
0926 21:36:37.856525 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 21:36:37.856938 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":730,"threads_started":1}
I20260926 21:36:37.859871 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42915
I20260926 21:36:37.860294 896 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:37.861231 896 heartbeater.cc:511] Master 127.0.97.62:42915 requested a full tablet report, sending...
I20260926 21:36:37.862658 440 ts_manager.cc:194] Registered new tserver with Master: 17fa0db9e50242d191a9d0a9ce20aec1 (127.0.97.3:34161)
I20260926 21:36:37.863413 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38439
I20260926 21:36:37.873981 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:36:37 dist-test-slave-lhhq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:36:37 dist-test-slave-lhhq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:36:37.918792 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:37.903574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.0.97.1:40741)
0926 21:36:37.904222 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:37.904229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:37.904256 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:36:37.904359 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:37.904365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:37.904585 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:37.904717 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:37.906237 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.907233 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.908491 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.909018 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.909135 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:37.913070 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:37.913082 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:37.913096 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:37.913135 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:37.916340 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.916987 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.916994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.916997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.917112 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.917515 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.917518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.917520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.917843 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:37.918063 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:37.918313 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 21:36:37.918493 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":328,"threads_started":1}
W20260926 21:36:37.920120 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59464
Sep 26 21:36:37 dist-test-slave-lhhq krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 21:36:37 dist-test-slave-lhhq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:36:37.964259 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:37.950201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.0.97.1:40741)
0926 21:36:37.950435 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:37.950443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:37.950712 (+ 269us) server_negotiation.cc:407] Connection header received
0926 21:36:37.950946 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:37.950951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:37.951029 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:37.951140 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:37.952476 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.953163 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.954458 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.954898 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.955035 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:37.958900 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:37.958907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:37.958914 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:37.958962 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:37.961354 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.961980 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.961986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.961990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.962118 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.962476 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.962483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.962486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.962745 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:37.962982 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:37.963293 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 21:36:37.963907 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:36:37.965839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 501
I20260926 21:36:37.978598 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:37.966750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.0.97.1:40741)
0926 21:36:37.966869 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:37.966873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:37.967267 (+ 394us) server_negotiation.cc:407] Connection header received
0926 21:36:37.967448 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:37.967453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:37.967519 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:37.967619 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:37.968773 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.969542 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.970814 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:37.971204 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:37.971341 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:37.973563 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:37.973567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:37.973570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:37.973599 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:37.975949 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.976598 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.976605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.976609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.976770 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:37.977138 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:37.977143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:37.977146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:37.977491 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:37.977741 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:37.978120 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 21:36:37.978285 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 21:36:37.979389 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:38.063938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 501
I20260926 21:36:38.080090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 635
Sep 26 21:36:38 dist-test-slave-lhhq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:36:38.096102 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:38.081002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.0.97.2:33951)
0926 21:36:38.081523 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:38.081531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:38.081557 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:36:38.081629 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:38.081636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:38.081847 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:38.081973 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:38.083118 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:38.084137 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:38.085424 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:38.085968 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:38.086067 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:38.090491 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:38.090504 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:38.090519 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:38.090556 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:38.093590 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:38.094233 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:38.094240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:38.094243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:38.094366 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:38.094687 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:38.094690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:38.094692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:38.095023 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:38.095258 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:38.095514 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 21:36:38.095713 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":289,"threads_started":1}
I20260926 21:36:38.097553 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:38.179471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 635
I20260926 21:36:38.201438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 768
Sep 26 21:36:38 dist-test-slave-lhhq krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:36:38.217629 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:38.202556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.97.3:34161)
0926 21:36:38.203222 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:38.203229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:38.203259 (+ 30us) server_negotiation.cc:407] Connection header received
0926 21:36:38.203317 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:38.203323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:38.203530 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:38.203668 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:38.204844 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:38.205927 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:38.207180 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:38.207737 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:38.207833 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:38.212004 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:38.212017 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:38.212032 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:38.212069 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:38.215239 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:38.215869 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:38.215876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:38.215878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:38.215996 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:38.216308 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:38.216311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:38.216313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:38.216645 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:38.216883 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:38.217082 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 21:36:38.217343 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":350,"threads_started":1}
I20260926 21:36:38.218933 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:38.300761 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 768
I20260926 21:36:38.317262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 408
I20260926 21:36:38.318399 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:38.421039 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 408
2026-09-26T21:36:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5115 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:36:38 dist-test-slave-lhhq krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:36:38 dist-test-slave-lhhq krb5kdc[918](info): set up 2 sockets
Sep 26 21:36:38 dist-test-slave-lhhq krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260926 21:36:39.497574 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:36:39 dist-test-slave-lhhq krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:36:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:36:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:36:39.681751 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36393
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:40.003687 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:40.004060 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:40.004220 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:40.011729 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:36:40.011904 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:40.012028 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:40.012133 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:36:40.012229 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:36:40.022334 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:40.023698 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:40.025043 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:40.036993 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:40.037073 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:40.037029 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:40.042929 931 server_base.cc:1034] running on GCE node
I20260926 21:36:40.043835 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:40.045392 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:40.046602 931 hybrid_clock.cc:648] HybridClock initialized: now 1790458600046555 us; error 33 us; skew 500 ppm
I20260926 21:36:40.046965 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:40 dist-test-slave-lhhq krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:40.058145 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:36:40.059731 931 webserver.cc:460] Webserver started at http://127.0.97.62:42125/ using document root <none> and password file <none>
I20260926 21:36:40.060436 931 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:40.060552 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:40.060827 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:40.063856 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "652c87776d1744268b12bb8f90a0b540"
format_stamp: "Formatted at 2026-09-26 21:36:40 on dist-test-slave-lhhq"
server_key: "3ffc3312182deb4fd0292afd0a92fb63"
server_key_iv: "c0c8047d5ad50bfd8df5a3598bd0d75b"
server_key_version: "encryptionkey@0"
I20260926 21:36:40.064553 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "652c87776d1744268b12bb8f90a0b540"
format_stamp: "Formatted at 2026-09-26 21:36:40 on dist-test-slave-lhhq"
server_key: "3ffc3312182deb4fd0292afd0a92fb63"
server_key_iv: "c0c8047d5ad50bfd8df5a3598bd0d75b"
server_key_version: "encryptionkey@0"
I20260926 21:36:40.069888 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:36:40.073452 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:40.074839 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:36:40.075033 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "652c87776d1744268b12bb8f90a0b540"
format_stamp: "Formatted at 2026-09-26 21:36:40 on dist-test-slave-lhhq"
server_key: "3ffc3312182deb4fd0292afd0a92fb63"
server_key_iv: "c0c8047d5ad50bfd8df5a3598bd0d75b"
server_key_version: "encryptionkey@0"
I20260926 21:36:40.075246 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:40.092365 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:40.097016 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:40.097386 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:40.115962 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36393
I20260926 21:36:40.115991 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36393 every 8 connection(s)
I20260926 21:36:40.117484 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:36:40.119218 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 931
I20260926 21:36:40.119426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:36:40.120251 388 external_mini_cluster.cc:1428] Setting key 15d619383207c165fa0300d720b8d149
I20260926 21:36:40.122594 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:36:40 dist-test-slave-lhhq krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:40.137633 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540: Bootstrap starting.
I20260926 21:36:40.141964 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:40.143046 998 log.cc:824] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:40.146116 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540: No bootstrap required, opened a new log
I20260926 21:36:40.146416 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:40.122345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.0.97.62:36393)
0926 21:36:40.123224 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:40.123236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:40.123276 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:36:40.124923 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:40.124944 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:40.125432 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:40.125775 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:40.127569 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:40.128933 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:40.130664 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:40.131691 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:40.131881 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:40.137975 (+ 6094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:40.137988 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:40.138005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:40.138081 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:40.142024 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:40.142841 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:40.142851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:40.142860 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:40.143052 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:40.143446 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:40.143451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:40.143454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:40.144031 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:40.144360 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:40.144871 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 21:36:40.145360 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":399,"threads_started":1}
I20260926 21:36:40.154320 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652c87776d1744268b12bb8f90a0b540" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36393 } }
I20260926 21:36:40.154769 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:40.154881 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652c87776d1744268b12bb8f90a0b540, State: Initialized, Role: FOLLOWER
I20260926 21:36:40.155462 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652c87776d1744268b12bb8f90a0b540" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36393 } }
I20260926 21:36:40.155733 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:36:40.155880 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:36:40.156056 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:40.158298 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652c87776d1744268b12bb8f90a0b540" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36393 } }
I20260926 21:36:40.158821 998 leader_election.cc:302] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 652c87776d1744268b12bb8f90a0b540; no voters:
I20260926 21:36:40.159395 998 leader_election.cc:288] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:36:40.159528 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:40.160568 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [term 1 LEADER]: Becoming Leader. State: Replica: 652c87776d1744268b12bb8f90a0b540, State: Running, Role: LEADER
I20260926 21:36:40.161443 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652c87776d1744268b12bb8f90a0b540" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36393 } }
I20260926 21:36:40.161648 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:36:40.164285 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 652c87776d1744268b12bb8f90a0b540. Latest consensus state: current_term: 1 leader_uuid: "652c87776d1744268b12bb8f90a0b540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652c87776d1744268b12bb8f90a0b540" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36393 } } }
I20260926 21:36:40.164722 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:40.164359 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "652c87776d1744268b12bb8f90a0b540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652c87776d1744268b12bb8f90a0b540" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36393 } } }
I20260926 21:36:40.164904 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540 [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:40.166652 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:36:40.171977 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:36:40.181473 1012 catalog_manager.cc:1347] Generated new cluster ID: a7f7b8efcea24d94b52f13ac5e75838e
I20260926 21:36:40.181612 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:36:40.196442 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:36:40.197942 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:36:40.206602 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 652c87776d1744268b12bb8f90a0b540: Generated new TSK 0
I20260926 21:36:40.207526 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:36:40.260749 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36393
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:40.607918 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:40.608309 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:40.608412 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:40.615525 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:40.615628 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:40.615742 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:36:40.625351 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36393
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:40.626588 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:40.627867 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:40.640671 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:40.640771 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:40.640722 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:40.641530 1024 server_base.cc:1034] running on GCE node
I20260926 21:36:40.642175 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:40.643108 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:40.644340 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790458600644288 us; error 45 us; skew 500 ppm
I20260926 21:36:40.644676 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:40 dist-test-slave-lhhq krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:40.647791 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:36:40.649296 1024 webserver.cc:460] Webserver started at http://127.0.97.1:36207/ using document root <none> and password file <none>
I20260926 21:36:40.649972 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:40.650120 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:40.650429 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:40.653350 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "35fc788ee845447bbd8dc0853b6f1fc4"
format_stamp: "Formatted at 2026-09-26 21:36:40 on dist-test-slave-lhhq"
server_key: "db67c7e5346be4173117a72e2c250594"
server_key_iv: "e9af7583d33de41ed5e0b7979aed49ae"
server_key_version: "encryptionkey@0"
I20260926 21:36:40.654114 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "35fc788ee845447bbd8dc0853b6f1fc4"
format_stamp: "Formatted at 2026-09-26 21:36:40 on dist-test-slave-lhhq"
server_key: "db67c7e5346be4173117a72e2c250594"
server_key_iv: "e9af7583d33de41ed5e0b7979aed49ae"
server_key_version: "encryptionkey@0"
I20260926 21:36:40.659003 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:36:40.662357 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:40.663780 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:36:40.663964 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "35fc788ee845447bbd8dc0853b6f1fc4"
format_stamp: "Formatted at 2026-09-26 21:36:40 on dist-test-slave-lhhq"
server_key: "db67c7e5346be4173117a72e2c250594"
server_key_iv: "e9af7583d33de41ed5e0b7979aed49ae"
server_key_version: "encryptionkey@0"
I20260926 21:36:40.664203 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:40.691955 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:40.696149 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:40.696462 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:40.697492 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:36:40.698925 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:40.699036 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:40.699200 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:40.699275 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:40.733090 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45993
I20260926 21:36:40.733148 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45993 every 8 connection(s)
I20260926 21:36:40.734493 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:36:40.740818 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 1024
I20260926 21:36:40.741039 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:36:40.741869 388 external_mini_cluster.cc:1428] Setting key f14dedcf1e41ce3d1b3d8d04060f2fbe
Sep 26 21:36:40 dist-test-slave-lhhq krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:40.759091 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:40.737326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47285 (local address 127.0.97.62:36393)
0926 21:36:40.737782 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:40.737786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:40.738703 (+ 917us) server_negotiation.cc:407] Connection header received
0926 21:36:40.740697 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:40.740702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:40.740812 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:40.740914 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:40.743060 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:40.744108 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:40.746855 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:40.747252 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:40.747359 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:40.751739 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:40.751748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:40.751757 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:40.751804 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:40.755465 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:40.756430 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:40.756436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:40.756441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:40.756601 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:40.757049 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:40.757056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:40.757061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:40.757507 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:40.757704 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:40.758389 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 21:36:40.758636 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":266,"threads_started":1}
I20260926 21:36:40.759824 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:40.737808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36393 (local address 127.0.97.1:47285)
0926 21:36:40.738531 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:40.738574 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:40.740323 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:40.741264 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:40.741280 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:40.742081 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:40.742870 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:40.742893 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:40.744279 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:40.744287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:40.746666 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:40.746693 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:40.747497 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:40.747506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:40.747858 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:40.748630 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:40.748663 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:40.751451 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:40.755644 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:40.755652 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:40.755671 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:40.756271 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:40.756729 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:40.756732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:40.756734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:40.756921 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:40.757636 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:40.757642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:40.757980 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 21:36:40.758216 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 21:36:40.758617 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:36:40.761585 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36393
I20260926 21:36:40.761875 1152 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:40.762771 1152 heartbeater.cc:511] Master 127.0.97.62:36393 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:36:40.765174 963 ts_manager.cc:194] Registered new tserver with Master: 35fc788ee845447bbd8dc0853b6f1fc4 (127.0.97.1:45993)
I20260926 21:36:40.766913 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:36:40.785444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36393
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:41.114950 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:41.115319 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:41.115470 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:41.127013 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:41.127168 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:41.127346 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:36:41.141573 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36393
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:41.142819 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:41.144112 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:41.157522 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:41.157545 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:41.157583 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:41.158365 1158 server_base.cc:1034] running on GCE node
I20260926 21:36:41.158952 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:41.159869 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:41.161062 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790458601161007 us; error 44 us; skew 500 ppm
I20260926 21:36:41.161437 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:41 dist-test-slave-lhhq krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:41.164620 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:36:41.166203 1158 webserver.cc:460] Webserver started at http://127.0.97.2:40137/ using document root <none> and password file <none>
I20260926 21:36:41.166941 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:41.167074 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:41.167409 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:41.170352 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "c6bcebacd732492f8f163cbca04e3360"
format_stamp: "Formatted at 2026-09-26 21:36:41 on dist-test-slave-lhhq"
server_key: "6ca4b8021f0585e28982947c1fac30f5"
server_key_iv: "39f3b17aef23adb9dc02422f71e43d16"
server_key_version: "encryptionkey@0"
I20260926 21:36:41.171082 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c6bcebacd732492f8f163cbca04e3360"
format_stamp: "Formatted at 2026-09-26 21:36:41 on dist-test-slave-lhhq"
server_key: "6ca4b8021f0585e28982947c1fac30f5"
server_key_iv: "39f3b17aef23adb9dc02422f71e43d16"
server_key_version: "encryptionkey@0"
I20260926 21:36:41.176355 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:36:41.180038 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:41.181684 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:36:41.181870 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "c6bcebacd732492f8f163cbca04e3360"
format_stamp: "Formatted at 2026-09-26 21:36:41 on dist-test-slave-lhhq"
server_key: "6ca4b8021f0585e28982947c1fac30f5"
server_key_iv: "39f3b17aef23adb9dc02422f71e43d16"
server_key_version: "encryptionkey@0"
I20260926 21:36:41.182121 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:41.196220 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:41.200333 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:41.200577 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:41.201502 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:36:41.202826 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:41.202898 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:41.202981 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:41.203033 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:41.236464 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34173
I20260926 21:36:41.236519 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34173 every 8 connection(s)
I20260926 21:36:41.237859 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:36:41.243515 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 1158
I20260926 21:36:41.243695 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:36:41.244115 388 external_mini_cluster.cc:1428] Setting key 468e9228352fafc8a3a8be5635861adf
Sep 26 21:36:41 dist-test-slave-lhhq krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:41.258087 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.240502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50847 (local address 127.0.97.62:36393)
0926 21:36:41.240684 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:41.240689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:41.241817 (+ 1128us) server_negotiation.cc:407] Connection header received
0926 21:36:41.243607 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:41.243612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:41.243706 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:41.243820 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:41.245532 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.246339 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.247946 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.248384 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.248524 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.252277 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:41.252282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:41.252286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:41.252318 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:41.254967 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:41.255727 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:41.255735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:41.255740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:41.255882 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:41.256318 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:41.256325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:41.256333 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:41.256588 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:41.256742 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:41.257511 (+ 769us) server_negotiation.cc:299] Negotiation successful
0926 21:36:41.257745 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:36:41.259239 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.240931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36393 (local address 127.0.97.2:50847)
0926 21:36:41.241647 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:41.241694 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:41.243338 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:41.244051 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:41.244059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:41.244612 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:41.245368 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.245387 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.246485 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.246495 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.247787 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.247800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.248611 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.248618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.248862 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.249592 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:41.249614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:41.252020 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:41.255122 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:41.255132 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:41.255151 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:41.255592 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:41.255997 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:41.256002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:41.256006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:41.256197 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:41.256726 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:41.256734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:41.257077 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 21:36:41.257413 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 21:36:41.257976 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":311,"threads_started":1}
I20260926 21:36:41.260979 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36393
I20260926 21:36:41.261406 1286 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 21:36:41.262262 1286 heartbeater.cc:511] Master 127.0.97.62:36393 requested a full tablet report, sending...
I20260926 21:36:41.264119 963 ts_manager.cc:194] Registered new tserver with Master: c6bcebacd732492f8f163cbca04e3360 (127.0.97.2:34173)
I20260926 21:36:41.265432 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:36:41.284044 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36393
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:41.609151 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:41.609546 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:41.609731 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:41.617128 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:41.617339 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:41.617537 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:36:41.627532 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36393
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:41.628949 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:41.630435 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:41.644332 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:41.644492 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:41.644328 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:41.645412 1291 server_base.cc:1034] running on GCE node
I20260926 21:36:41.646034 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:41.647166 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:41.648423 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790458601648386 us; error 61 us; skew 500 ppm
I20260926 21:36:41.648773 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:41 dist-test-slave-lhhq krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:41.651919 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:36:41.653579 1291 webserver.cc:460] Webserver started at http://127.0.97.3:40637/ using document root <none> and password file <none>
I20260926 21:36:41.654234 1291 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:41.654333 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:41.654578 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:41.657420 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "1105ede748f34280a4c8c06b8da37745"
format_stamp: "Formatted at 2026-09-26 21:36:41 on dist-test-slave-lhhq"
server_key: "634429e101f01185bc3391fc3093c971"
server_key_iv: "f34a0a9f1322b8184c0164e76038cf26"
server_key_version: "encryptionkey@0"
I20260926 21:36:41.658152 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1105ede748f34280a4c8c06b8da37745"
format_stamp: "Formatted at 2026-09-26 21:36:41 on dist-test-slave-lhhq"
server_key: "634429e101f01185bc3391fc3093c971"
server_key_iv: "f34a0a9f1322b8184c0164e76038cf26"
server_key_version: "encryptionkey@0"
I20260926 21:36:41.663444 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:36:41.666949 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:41.668401 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:36:41.668532 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "1105ede748f34280a4c8c06b8da37745"
format_stamp: "Formatted at 2026-09-26 21:36:41 on dist-test-slave-lhhq"
server_key: "634429e101f01185bc3391fc3093c971"
server_key_iv: "f34a0a9f1322b8184c0164e76038cf26"
server_key_version: "encryptionkey@0"
I20260926 21:36:41.668701 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:41.699868 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:41.703886 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:41.704125 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:41.705051 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:36:41.706455 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:41.706526 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:41.706609 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:41.706657 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:41.743896 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41411
I20260926 21:36:41.743937 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41411 every 8 connection(s)
I20260926 21:36:41.745225 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:36:41.751816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 1291
I20260926 21:36:41.751998 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:36:41.752595 388 external_mini_cluster.cc:1428] Setting key 496e03cb2bda3baf9619bbd61ab9e35b
Sep 26 21:36:41 dist-test-slave-lhhq krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:41.766206 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.747973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57905 (local address 127.0.97.62:36393)
0926 21:36:41.748206 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:41.748212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:41.749435 (+ 1223us) server_negotiation.cc:407] Connection header received
0926 21:36:41.751245 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:41.751250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:41.751317 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:41.751466 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:41.753116 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.753951 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.756044 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.756418 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.756511 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.759981 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:41.759990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:41.759994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:41.760037 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:41.763026 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:41.763719 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:41.763724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:41.763726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:41.763853 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:41.764285 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:41.764289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:41.764291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:41.764562 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:41.764704 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:41.765608 (+ 904us) server_negotiation.cc:299] Negotiation successful
0926 21:36:41.765847 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 21:36:41.767769 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.748469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36393 (local address 127.0.97.3:57905)
0926 21:36:41.749249 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:41.749307 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:41.750962 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:41.751676 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:41.751685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:41.752303 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:41.752928 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.752948 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.754119 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.754123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.755911 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.755922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.756603 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.756607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.756817 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.757503 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:41.757525 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:41.759729 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:41.763164 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:41.763172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:41.763186 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:41.763605 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:41.763954 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:41.763958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:41.763960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:41.764163 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:41.764720 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:41.764727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:41.765101 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 21:36:41.765443 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 21:36:41.765970 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":288,"threads_started":1}
I20260926 21:36:41.769287 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36393
I20260926 21:36:41.769604 1419 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:41.770421 1419 heartbeater.cc:511] Master 127.0.97.62:36393 requested a full tablet report, sending...
I20260926 21:36:41.770423 1152 heartbeater.cc:503] Master 127.0.97.62:36393 was elected leader, sending a full tablet report...
I20260926 21:36:41.772063 962 ts_manager.cc:194] Registered new tserver with Master: 1105ede748f34280a4c8c06b8da37745 (127.0.97.3:41411)
I20260926 21:36:41.772899 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57905
I20260926 21:36:41.780934 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:36:41.801178 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.787538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.0.97.62:36393)
0926 21:36:41.787740 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:41.787744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:41.788306 (+ 562us) server_negotiation.cc:407] Connection header received
0926 21:36:41.788480 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:41.788484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:41.788548 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:41.788639 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:41.790362 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.791103 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.792429 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.792834 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.792961 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.795893 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:41.795899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:41.795903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:41.795944 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:41.798642 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:41.799349 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:41.799354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:41.799356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:41.799492 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:41.799874 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:41.799877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:41.799879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:41.800238 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:41.800421 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:41.800761 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 21:36:41.800937 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 21:36:41.808058 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:36:41.815245 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:36:41.850454 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.829210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.0.97.1:45993)
0926 21:36:41.842354 (+ 13144us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:41.842363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:41.842392 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:36:41.842493 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:41.842501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:41.842779 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:41.843675 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:41.844475 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.845925 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.847918 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.849243 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.849416 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.849663 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:41.849934 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 21:36:41.850119 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12975,"thread_start_us":12804,"threads_started":1}
I20260926 21:36:41.850533 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.828645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45993 (local address 127.0.0.1:38460)
0926 21:36:41.840643 (+ 11998us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:41.840682 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:41.840945 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:41.843021 (+ 2076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:41.843026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:41.843056 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:41.843629 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.843646 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.846051 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.846058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.847754 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.847766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.849458 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.849464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.849669 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.849725 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 21:36:41.849873 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 21:36:41.850082 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11764,"mutex_wait_us":43,"thread_start_us":13600,"threads_started":1}
I20260926 21:36:41.851240 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.829446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34173 (local address 127.0.0.1:54148)
0926 21:36:41.842392 (+ 12946us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:41.842415 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:41.842550 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:41.843307 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:41.843312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:41.843336 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:41.844233 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.844245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.846081 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.846085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.848310 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.848321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.850380 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.850387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.850602 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.850662 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 21:36:41.850783 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 21:36:41.850911 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12826,"thread_start_us":10771,"threads_started":1}
I20260926 21:36:41.851351 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.829751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.0.97.2:34173)
0926 21:36:41.841560 (+ 11809us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:41.841575 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:41.842547 (+ 972us) server_negotiation.cc:407] Connection header received
0926 21:36:41.842677 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:41.842683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:41.842868 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:41.843046 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:41.844428 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.845920 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.848471 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.849927 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.850060 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.850238 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:41.850840 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 21:36:41.851018 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11625,"thread_start_us":11482,"threads_started":1}
I20260926 21:36:41.852474 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.828289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.0.97.3:41411)
0926 21:36:41.842140 (+ 13851us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:41.842151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:41.842197 (+ 46us) server_negotiation.cc:407] Connection header received
0926 21:36:41.842313 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:41.842322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:41.842644 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:41.842874 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:41.843589 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.845473 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.849363 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.851092 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.851238 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.851464 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:41.851968 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 21:36:41.852151 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13673,"thread_start_us":13509,"threads_started":1}
I20260926 21:36:41.853691 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.827982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41411 (local address 127.0.0.1:38432)
0926 21:36:41.828844 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:41.828902 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:41.829151 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:41.842888 (+ 13737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:41.842897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:41.842932 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:41.843394 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.843405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.845666 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.845672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.849135 (+ 3463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.849155 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.851622 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.851629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.851787 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.851810 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:36:41.851924 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 21:36:41.853346 (+ 1422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":115,"thread_start_us":622,"threads_started":1}
I20260926 21:36:41.854954 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 08967f94b9944eb7866277dd165e42a3 (DEFAULT_TABLE table=test-table [id=2287e64a5d7d40d99a37306fa91d4854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:41.855768 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 08967f94b9944eb7866277dd165e42a3 (DEFAULT_TABLE table=test-table [id=2287e64a5d7d40d99a37306fa91d4854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:41.856599 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08967f94b9944eb7866277dd165e42a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:41.856838 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08967f94b9944eb7866277dd165e42a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:41.857662 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 08967f94b9944eb7866277dd165e42a3 (DEFAULT_TABLE table=test-table [id=2287e64a5d7d40d99a37306fa91d4854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:41.858990 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08967f94b9944eb7866277dd165e42a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:41.874482 1439 tablet_bootstrap.cc:492] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745: Bootstrap starting.
I20260926 21:36:41.874619 1438 tablet_bootstrap.cc:492] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4: Bootstrap starting.
I20260926 21:36:41.876431 1437 tablet_bootstrap.cc:492] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360: Bootstrap starting.
I20260926 21:36:41.879766 1437 tablet_bootstrap.cc:654] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:41.879784 1439 tablet_bootstrap.cc:654] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:41.880364 1438 tablet_bootstrap.cc:654] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:41.881058 1437 log.cc:824] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:41.881584 1439 log.cc:824] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:41.881582 1438 log.cc:824] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:41.883970 1437 tablet_bootstrap.cc:492] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360: No bootstrap required, opened a new log
I20260926 21:36:41.884246 1437 ts_tablet_manager.cc:1397] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 21:36:41.884959 1439 tablet_bootstrap.cc:492] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745: No bootstrap required, opened a new log
I20260926 21:36:41.885263 1439 ts_tablet_manager.cc:1397] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 21:36:41.887437 1438 tablet_bootstrap.cc:492] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4: No bootstrap required, opened a new log
I20260926 21:36:41.887703 1438 ts_tablet_manager.cc:1397] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20260926 21:36:41.892702 1437 raft_consensus.cc:348] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } }
I20260926 21:36:41.893113 1437 raft_consensus.cc:374] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:41.893249 1437 raft_consensus.cc:729] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcebacd732492f8f163cbca04e3360, State: Initialized, Role: FOLLOWER
I20260926 21:36:41.893873 1437 consensus_queue.cc:260] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } }
I20260926 21:36:41.894972 1286 heartbeater.cc:503] Master 127.0.97.62:36393 was elected leader, sending a full tablet report...
I20260926 21:36:41.896776 1437 ts_tablet_manager.cc:1428] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260926 21:36:41.896236 1439 raft_consensus.cc:348] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } }
I20260926 21:36:41.897269 1439 raft_consensus.cc:374] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:41.897406 1439 raft_consensus.cc:729] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1105ede748f34280a4c8c06b8da37745, State: Initialized, Role: FOLLOWER
I20260926 21:36:41.898031 1439 consensus_queue.cc:260] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } }
I20260926 21:36:41.898042 1438 raft_consensus.cc:348] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } }
I20260926 21:36:41.898496 1438 raft_consensus.cc:374] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:41.898598 1438 raft_consensus.cc:729] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35fc788ee845447bbd8dc0853b6f1fc4, State: Initialized, Role: FOLLOWER
I20260926 21:36:41.899634 1438 consensus_queue.cc:260] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } }
I20260926 21:36:41.901608 1438 ts_tablet_manager.cc:1428] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260926 21:36:41.901942 1419 heartbeater.cc:503] Master 127.0.97.62:36393 was elected leader, sending a full tablet report...
I20260926 21:36:41.902565 1439 ts_tablet_manager.cc:1428] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
I20260926 21:36:41.934515 1444 raft_consensus.cc:482] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:36:41.934803 1444 raft_consensus.cc:504] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } }
I20260926 21:36:41.936501 1444 leader_election.cc:288] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6bcebacd732492f8f163cbca04e3360 (127.0.97.2:34173), 35fc788ee845447bbd8dc0853b6f1fc4 (127.0.97.1:45993)
I20260926 21:36:41.944342 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.936899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57871 (local address 127.0.97.2:34173)
0926 21:36:41.937121 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:41.937127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:41.937518 (+ 391us) server_negotiation.cc:407] Connection header received
0926 21:36:41.937744 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:41.937749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:41.937839 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:41.937935 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:41.938452 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.939336 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.941861 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.943488 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.943631 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.943728 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:41.943987 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 21:36:41.944079 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:36:41.944401 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.936956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34173 (local address 127.0.97.3:57871)
0926 21:36:41.937373 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:41.937401 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:41.937571 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:41.937993 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:41.937998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:41.938026 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:41.938278 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.938291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.939503 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.939509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.941709 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.941724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.943646 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.943650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.943779 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.943836 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 21:36:41.943944 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 21:36:41.944083 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":37}
I20260926 21:36:41.944698 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.937145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59227 (local address 127.0.97.1:45993)
0926 21:36:41.937410 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:41.937418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:41.937884 (+ 466us) server_negotiation.cc:407] Connection header received
0926 21:36:41.937995 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:41.938001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:41.938074 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:41.938190 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:41.938972 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.939810 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.942141 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.943722 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.943836 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.943926 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:41.944248 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 21:36:41.944393 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 21:36:41.945152 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.936987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45993 (local address 127.0.97.3:59227)
0926 21:36:41.937750 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:41.937767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:41.937895 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:41.938242 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:41.938246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:41.938266 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:41.938638 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.938650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.939990 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.939999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.942001 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:41.942010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.943926 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.943935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:41.944089 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.944117 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 21:36:41.944202 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 21:36:41.944318 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":18,"thread_start_us":453,"threads_started":1}
I20260926 21:36:41.945524 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08967f94b9944eb7866277dd165e42a3" candidate_uuid: "1105ede748f34280a4c8c06b8da37745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcebacd732492f8f163cbca04e3360" is_pre_election: true
I20260926 21:36:41.946077 1241 raft_consensus.cc:2454] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1105ede748f34280a4c8c06b8da37745 in term 0.
I20260926 21:36:41.946246 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08967f94b9944eb7866277dd165e42a3" candidate_uuid: "1105ede748f34280a4c8c06b8da37745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" is_pre_election: true
I20260926 21:36:41.946740 1107 raft_consensus.cc:2454] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1105ede748f34280a4c8c06b8da37745 in term 0.
I20260926 21:36:41.946755 1308 leader_election.cc:302] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1105ede748f34280a4c8c06b8da37745, c6bcebacd732492f8f163cbca04e3360; no voters:
I20260926 21:36:41.947132 1444 raft_consensus.cc:2781] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:36:41.947292 1444 raft_consensus.cc:482] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:36:41.947402 1444 raft_consensus.cc:3037] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:41.949692 1444 raft_consensus.cc:504] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } }
I20260926 21:36:41.950287 1444 leader_election.cc:288] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [CANDIDATE]: Term 1 election: Requested vote from peers c6bcebacd732492f8f163cbca04e3360 (127.0.97.2:34173), 35fc788ee845447bbd8dc0853b6f1fc4 (127.0.97.1:45993)
I20260926 21:36:41.950646 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08967f94b9944eb7866277dd165e42a3" candidate_uuid: "1105ede748f34280a4c8c06b8da37745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcebacd732492f8f163cbca04e3360"
I20260926 21:36:41.950743 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08967f94b9944eb7866277dd165e42a3" candidate_uuid: "1105ede748f34280a4c8c06b8da37745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fc788ee845447bbd8dc0853b6f1fc4"
I20260926 21:36:41.950851 1241 raft_consensus.cc:3037] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:41.950953 1107 raft_consensus.cc:3037] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:41.953162 1107 raft_consensus.cc:2454] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1105ede748f34280a4c8c06b8da37745 in term 1.
I20260926 21:36:41.953306 1241 raft_consensus.cc:2454] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1105ede748f34280a4c8c06b8da37745 in term 1.
I20260926 21:36:41.953605 1309 leader_election.cc:302] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1105ede748f34280a4c8c06b8da37745, 35fc788ee845447bbd8dc0853b6f1fc4; no voters:
I20260926 21:36:41.953909 1444 raft_consensus.cc:2781] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:41.954351 1444 raft_consensus.cc:686] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [term 1 LEADER]: Becoming Leader. State: Replica: 1105ede748f34280a4c8c06b8da37745, State: Running, Role: LEADER
I20260926 21:36:41.954881 1444 consensus_queue.cc:237] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } }
I20260926 21:36:41.959717 962 catalog_manager.cc:5495] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1105ede748f34280a4c8c06b8da37745 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1105ede748f34280a4c8c06b8da37745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1105ede748f34280a4c8c06b8da37745" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bcebacd732492f8f163cbca04e3360" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35fc788ee845447bbd8dc0853b6f1fc4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45993 } health_report { overall_health: UNKNOWN } } }
Sep 26 21:36:41 dist-test-slave-lhhq krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:36:41.982618 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:41.966683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.0.97.1:45993)
0926 21:36:41.966837 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:41.966841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:41.967485 (+ 644us) server_negotiation.cc:407] Connection header received
0926 21:36:41.967721 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:41.967725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:41.967830 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:41.967932 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:41.969496 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.970218 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.971366 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:41.971802 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:41.971927 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:41.975935 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:41.975948 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:41.975962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:41.975994 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:41.979067 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:41.979936 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:41.979943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:41.979946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:41.980066 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:41.980554 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:41.980558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:41.980560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:41.981005 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:41.981333 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:41.982007 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 21:36:41.982232 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 21:36:41.986981 1153 tablet.cc:2369] T 08967f94b9944eb7866277dd165e42a3 P 35fc788ee845447bbd8dc0853b6f1fc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:36:41.989863 1287 tablet.cc:2369] T 08967f94b9944eb7866277dd165e42a3 P c6bcebacd732492f8f163cbca04e3360: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:36:41.996906 1420 tablet.cc:2369] T 08967f94b9944eb7866277dd165e42a3 P 1105ede748f34280a4c8c06b8da37745: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 21:36:42 dist-test-slave-lhhq krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:36:42 dist-test-slave-lhhq krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:36:42.029002 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:42.015233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.0.97.1:45993)
0926 21:36:42.015492 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:42.015500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:42.015811 (+ 311us) server_negotiation.cc:407] Connection header received
0926 21:36:42.016068 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:42.016072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:42.016142 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:42.016254 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:42.017615 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.018343 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.019646 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.020068 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.020200 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:42.023967 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:42.023972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:42.023976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:42.024009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:42.026895 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.027493 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.027497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.027499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.027615 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.027949 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.027952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.027954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.028203 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:42.028402 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:42.028588 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 21:36:42.028766 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 21:36:42 dist-test-slave-lhhq krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 21:36:42 dist-test-slave-lhhq krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:36:42.074618 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:42.059454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.0.97.1:45993)
0926 21:36:42.059673 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:42.059680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:42.060487 (+ 807us) server_negotiation.cc:407] Connection header received
0926 21:36:42.060784 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:42.060791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:42.060866 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:42.060975 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:42.062482 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.063206 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.064470 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.064888 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.065019 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:42.069050 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:42.069077 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:42.069083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:42.069128 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:42.071731 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.072512 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.072518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.072522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.072653 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.073036 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.073041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.073044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.073452 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:42.073702 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:42.074011 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 21:36:42.074269 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 21:36:42.076395 1067 tablet_service.cc:2908] Rejecting scan request for tablet 08967f94b9944eb7866277dd165e42a3: Uninitialized: clean time has not yet been initialized
I20260926 21:36:42.079499 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1024
I20260926 21:36:42.092326 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:42.080604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.0.97.1:45993)
0926 21:36:42.080771 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:42.080775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:42.081165 (+ 390us) server_negotiation.cc:407] Connection header received
0926 21:36:42.081361 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:42.081366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:42.081436 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:42.081517 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:42.082579 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.083243 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.084377 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.084787 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.084888 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:42.087549 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:42.087554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:42.087557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:42.087588 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:42.090216 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.090808 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.090812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.090814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.090929 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.091239 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.091242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.091244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.091499 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:42.091695 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:42.091929 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 21:36:42.092098 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:36:42.092963 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:42.181540 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 1024
I20260926 21:36:42.198884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1158
Sep 26 21:36:42 dist-test-slave-lhhq krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:36:42.213616 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:42.199627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.0.97.2:34173)
0926 21:36:42.199799 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:42.199805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:42.199823 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:36:42.199959 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:42.199963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:42.200059 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:42.200145 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:42.201264 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.202013 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.203259 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.203648 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.203771 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:42.207837 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:42.207852 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:42.207869 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:42.207905 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:42.210836 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.211714 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.211722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.211724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.211855 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.212201 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.212204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.212206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.212573 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:42.212818 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:42.213063 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 21:36:42.213303 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:36:42.214351 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:42.301649 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 1158
I20260926 21:36:42.317948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1291
Sep 26 21:36:42 dist-test-slave-lhhq krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:36:42.332679 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:42.318690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.0.97.3:41411)
0926 21:36:42.318865 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:42.318870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:42.318887 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:36:42.319025 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:42.319029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:42.319098 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:42.319188 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:42.320325 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.321046 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.322414 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:42.322877 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:42.322988 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:42.327084 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:42.327104 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:42.327121 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:42.327156 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:42.329978 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.330640 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.330649 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.330651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.330793 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:42.331203 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:42.331208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:42.331218 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:42.331539 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:42.331844 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:42.332102 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 21:36:42.332340 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 21:36:42.333637 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:42.423094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 1291
I20260926 21:36:42.439409 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 931
I20260926 21:36:42.440536 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:42.547204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 931
2026-09-26T21:36:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4127 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:36:42 dist-test-slave-lhhq krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:36:42 dist-test-slave-lhhq krb5kdc[1466](info): set up 2 sockets
Sep 26 21:36:42 dist-test-slave-lhhq krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260926 21:36:43.629169 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:36:43 dist-test-slave-lhhq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:36:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:36:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:36:43.820897 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42675
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:44.138244 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:44.138530 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:44.138705 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:44.145805 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:36:44.145906 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 21:36:44.146008 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:44.146117 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:44.146210 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:36:44.146304 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:36:44.156725 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42675
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:44.158174 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:44.159519 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:44.171408 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:44.171439 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:44.172118 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:44.172133 1479 server_base.cc:1034] running on GCE node
I20260926 21:36:44.173090 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:44.174490 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:44.175743 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790458604175669 us; error 54 us; skew 500 ppm
I20260926 21:36:44.176077 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:44 dist-test-slave-lhhq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:44.179272 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:36:44.180665 1479 webserver.cc:460] Webserver started at http://127.0.97.62:36793/ using document root <none> and password file <none>
I20260926 21:36:44.181453 1479 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:44.181577 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:44.181910 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:44.185384 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "a4392a8865234599a9725e8a35d9367f"
format_stamp: "Formatted at 2026-09-26 21:36:44 on dist-test-slave-lhhq"
server_key: "3da478bae55f3d87fda0700c1c00600b"
server_key_iv: "ecdbcd3ccfa75b900f7e043e283fb391"
server_key_version: "encryptionkey@0"
I20260926 21:36:44.186188 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "a4392a8865234599a9725e8a35d9367f"
format_stamp: "Formatted at 2026-09-26 21:36:44 on dist-test-slave-lhhq"
server_key: "3da478bae55f3d87fda0700c1c00600b"
server_key_iv: "ecdbcd3ccfa75b900f7e043e283fb391"
server_key_version: "encryptionkey@0"
I20260926 21:36:44.191300 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:36:44.194828 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:44.196201 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:36:44.196393 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "a4392a8865234599a9725e8a35d9367f"
format_stamp: "Formatted at 2026-09-26 21:36:44 on dist-test-slave-lhhq"
server_key: "3da478bae55f3d87fda0700c1c00600b"
server_key_iv: "ecdbcd3ccfa75b900f7e043e283fb391"
server_key_version: "encryptionkey@0"
I20260926 21:36:44.196607 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:44.211115 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:44.215149 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:44.215416 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:44.236688 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42675
I20260926 21:36:44.236730 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42675 every 8 connection(s)
I20260926 21:36:44.238194 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:36:44.241997 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:44.247907 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 1479
I20260926 21:36:44.248140 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:36:44.248961 388 external_mini_cluster.cc:1428] Setting key 178e5290cf7517add78a5a26362a4a21
I20260926 21:36:44.253522 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f: Bootstrap starting.
I20260926 21:36:44.257488 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:44.258704 1556 log.cc:824] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:44.262087 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f: No bootstrap required, opened a new log
Sep 26 21:36:44 dist-test-slave-lhhq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:44.270344 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4392a8865234599a9725e8a35d9367f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42675 } }
I20260926 21:36:44.270749 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:44.270871 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4392a8865234599a9725e8a35d9367f, State: Initialized, Role: FOLLOWER
I20260926 21:36:44.271516 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4392a8865234599a9725e8a35d9367f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42675 } }
I20260926 21:36:44.271790 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:36:44.271963 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:36:44.272190 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:44.273452 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:44.251135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.0.97.62:42675)
0926 21:36:44.251972 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:44.251984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:44.252027 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:36:44.253861 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:44.253886 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:44.254318 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:44.254657 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:44.256561 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.258053 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.259792 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.260575 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.260725 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:44.265522 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:44.265531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:44.265548 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:44.265589 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:44.269005 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:44.269770 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:44.269779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:44.269785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:44.269950 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:44.270324 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:44.270334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:44.270339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:44.271029 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:44.271336 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:44.271793 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 21:36:44.272249 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":405,"threads_started":1}
I20260926 21:36:44.274770 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4392a8865234599a9725e8a35d9367f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42675 } }
I20260926 21:36:44.275311 1556 leader_election.cc:302] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4392a8865234599a9725e8a35d9367f; no voters:
I20260926 21:36:44.275978 1556 leader_election.cc:288] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:36:44.276104 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:44.277170 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [term 1 LEADER]: Becoming Leader. State: Replica: a4392a8865234599a9725e8a35d9367f, State: Running, Role: LEADER
I20260926 21:36:44.277801 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4392a8865234599a9725e8a35d9367f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42675 } }
I20260926 21:36:44.278503 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:36:44.281071 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4392a8865234599a9725e8a35d9367f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4392a8865234599a9725e8a35d9367f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42675 } } }
I20260926 21:36:44.281544 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:44.281337 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4392a8865234599a9725e8a35d9367f. Latest consensus state: current_term: 1 leader_uuid: "a4392a8865234599a9725e8a35d9367f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4392a8865234599a9725e8a35d9367f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42675 } } }
I20260926 21:36:44.281703 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:44.285503 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:36:44.290346 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:36:44.298939 1566 catalog_manager.cc:1347] Generated new cluster ID: f873371bedad4e7db51356642c2e4394
I20260926 21:36:44.299062 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:36:44.312758 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:36:44.314718 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:36:44.332692 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f: Generated new TSK 0
I20260926 21:36:44.333714 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:36:44.379372 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42675
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:44.708050 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:44.708411 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:44.708612 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:44.716006 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:44.716209 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:44.716389 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:36:44.716547 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:36:44.726980 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:44.728688 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:44.730165 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:44.743412 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:44.743376 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:44.743450 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:44.744282 1582 server_base.cc:1034] running on GCE node
I20260926 21:36:44.744992 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:44.746058 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:44.747298 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790458604747238 us; error 42 us; skew 500 ppm
I20260926 21:36:44.747598 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:44 dist-test-slave-lhhq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:44.750797 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:36:44.752255 1582 webserver.cc:460] Webserver started at http://127.0.97.1:43663/ using document root <none> and password file <none>
I20260926 21:36:44.753026 1582 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:44.753152 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:44.753513 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:44.756402 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "d7cc9fa43b4045a28c208cacab6e8bbd"
format_stamp: "Formatted at 2026-09-26 21:36:44 on dist-test-slave-lhhq"
server_key: "2c50d82ea219650014e862f1a1cca86a"
server_key_iv: "a1739e3626347e852c16e4e9e7234526"
server_key_version: "encryptionkey@0"
I20260926 21:36:44.757124 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d7cc9fa43b4045a28c208cacab6e8bbd"
format_stamp: "Formatted at 2026-09-26 21:36:44 on dist-test-slave-lhhq"
server_key: "2c50d82ea219650014e862f1a1cca86a"
server_key_iv: "a1739e3626347e852c16e4e9e7234526"
server_key_version: "encryptionkey@0"
I20260926 21:36:44.762214 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:36:44.765566 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:44.767036 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:36:44.767207 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "d7cc9fa43b4045a28c208cacab6e8bbd"
format_stamp: "Formatted at 2026-09-26 21:36:44 on dist-test-slave-lhhq"
server_key: "2c50d82ea219650014e862f1a1cca86a"
server_key_iv: "a1739e3626347e852c16e4e9e7234526"
server_key_version: "encryptionkey@0"
I20260926 21:36:44.767416 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:44.795892 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:44.800594 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:44.800889 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:44.803493 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:44.803622 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:44.803758 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:44.803833 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:36:44 dist-test-slave-lhhq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:44.828143 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:44.804306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48925 (local address 127.0.97.62:42675)
0926 21:36:44.804780 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:44.804784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:44.806026 (+ 1242us) server_negotiation.cc:407] Connection header received
0926 21:36:44.808081 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:44.808087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:44.808175 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:44.808319 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:44.810705 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.811998 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.814526 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.815178 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.815323 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:44.820555 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:44.820564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:44.820570 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:44.820624 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:44.824142 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:44.825181 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:44.825213 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:44.825219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:44.825386 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:44.826070 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:44.826077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:44.826082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:44.826396 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:44.826592 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:44.827716 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0926 21:36:44.827886 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":272,"threads_started":1}
I20260926 21:36:44.830305 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:44.804970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42675 (local address 127.0.97.1:48925)
0926 21:36:44.805864 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:44.805906 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:44.807765 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:44.808644 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:44.808659 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:44.809588 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:44.810471 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:44.810510 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.812206 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.812213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:44.814369 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:44.814381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.815447 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.815455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:44.815738 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:44.816631 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:44.816657 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:44.820287 (+ 3630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:44.824349 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:44.824359 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:44.824378 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:44.825035 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:44.825525 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:44.825537 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:44.825543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:44.825907 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:44.826589 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:44.826597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:44.827183 (+ 586us) client_negotiation.cc:769] Sending connection context
0926 21:36:44.827608 (+ 425us) client_negotiation.cc:240] Negotiation successful
0926 21:36:44.828376 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":400,"threads_started":1}
I20260926 21:36:44.847858 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33023 every 8 connection(s)
I20260926 21:36:44.847858 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33023
I20260926 21:36:44.849628 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:36:44.850028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 1582
I20260926 21:36:44.850214 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:36:44.851044 388 external_mini_cluster.cc:1428] Setting key 067af20488334f2a3ec248db8be68240
I20260926 21:36:44.852578 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42675
I20260926 21:36:44.853229 1718 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:44.853911 1718 heartbeater.cc:511] Master 127.0.97.62:42675 requested a full tablet report, sending...
I20260926 21:36:44.856050 1511 ts_manager.cc:194] Registered new tserver with Master: d7cc9fa43b4045a28c208cacab6e8bbd (127.0.97.1:33023)
I20260926 21:36:44.857801 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48925
I20260926 21:36:44.863195 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:44.848184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42675 (local address 127.0.97.1:46869)
0926 21:36:44.848888 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:44.848904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:44.849027 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:44.849418 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:44.849421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:44.850041 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:44.850528 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:44.850561 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.852406 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.852413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:44.853852 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:44.853866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.854829 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.854839 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:44.855060 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:44.855777 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:44.855804 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:44.856883 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:44.860839 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:44.860843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:44.860845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:44.861309 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:44.861695 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:44.861698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:44.861700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:44.861822 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:44.862422 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:44.862425 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:44.862559 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 21:36:44.862667 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 21:36:44.862892 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":382,"threads_started":1}
I20260926 21:36:44.863462 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:44.848137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46869 (local address 127.0.97.62:42675)
0926 21:36:44.848352 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:44.848359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:44.849001 (+ 642us) server_negotiation.cc:407] Connection header received
0926 21:36:44.849146 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:44.849154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:44.849280 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:44.849394 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:44.850744 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.851827 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.854017 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:44.854609 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:44.854739 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:44.857073 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:44.857080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:44.857085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:44.857119 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:44.860706 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:44.861418 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:44.861426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:44.861433 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:44.861586 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:44.861929 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:44.861936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:44.861942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:44.862314 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:44.862463 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:44.862798 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 21:36:44.863051 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:36:44.890424 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42675
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:45.211333 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:45.211649 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:45.211823 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:45.219336 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:45.219485 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:45.219656 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:36:45.219794 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:36:45.229424 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:45.230580 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:45.231818 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:45.245508 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:45.245481 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:45.246239 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:45.247051 1722 server_base.cc:1034] running on GCE node
I20260926 21:36:45.247711 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:45.248713 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:45.249960 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790458605249894 us; error 43 us; skew 500 ppm
I20260926 21:36:45.250276 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:45 dist-test-slave-lhhq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:45.266515 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:36:45.268062 1722 webserver.cc:460] Webserver started at http://127.0.97.2:36631/ using document root <none> and password file <none>
I20260926 21:36:45.268731 1722 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:45.268829 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:45.269109 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:45.272209 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "39a77258e7ff44c9bff4db1c48176c6a"
format_stamp: "Formatted at 2026-09-26 21:36:45 on dist-test-slave-lhhq"
server_key: "80266067a906e81e041dc866c41f1a86"
server_key_iv: "c0f456973e8d8e90aae1bf2844b45418"
server_key_version: "encryptionkey@0"
I20260926 21:36:45.272976 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "39a77258e7ff44c9bff4db1c48176c6a"
format_stamp: "Formatted at 2026-09-26 21:36:45 on dist-test-slave-lhhq"
server_key: "80266067a906e81e041dc866c41f1a86"
server_key_iv: "c0f456973e8d8e90aae1bf2844b45418"
server_key_version: "encryptionkey@0"
I20260926 21:36:45.278162 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:36:45.281746 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:45.283041 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:36:45.283206 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "39a77258e7ff44c9bff4db1c48176c6a"
format_stamp: "Formatted at 2026-09-26 21:36:45 on dist-test-slave-lhhq"
server_key: "80266067a906e81e041dc866c41f1a86"
server_key_iv: "c0f456973e8d8e90aae1bf2844b45418"
server_key_version: "encryptionkey@0"
I20260926 21:36:45.283411 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:45.299815 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:45.303996 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:45.304283 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:45.306910 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:45.307019 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:45.307180 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:45.307257 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:36:45 dist-test-slave-lhhq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:45.331005 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.307665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34139 (local address 127.0.97.62:42675)
0926 21:36:45.307833 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:45.307839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:45.309372 (+ 1533us) server_negotiation.cc:407] Connection header received
0926 21:36:45.312432 (+ 3060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:45.312441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:45.312541 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:45.312660 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:45.315207 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.316196 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.318706 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.319224 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.319378 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.323873 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:45.323878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:45.323881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:45.323910 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:45.326913 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.327923 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.327929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.327933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.328106 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.328774 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.328780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.328783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.329224 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:45.329417 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:45.330395 (+ 978us) server_negotiation.cc:299] Negotiation successful
0926 21:36:45.330618 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:36:45.332582 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.308250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42675 (local address 127.0.97.2:34139)
0926 21:36:45.309171 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:45.309242 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:45.312077 (+ 2835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:45.312972 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:45.312990 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:45.313921 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:45.314987 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.315023 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.316368 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.316376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.318499 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.318520 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.319696 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.319704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.320001 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.320766 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:45.320811 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:45.323652 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:45.327087 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:45.327100 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:45.327117 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:45.327791 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:45.328236 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:45.328243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:45.328254 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:45.328589 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:45.329430 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:45.329444 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:45.329864 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 21:36:45.330290 (+ 426us) client_negotiation.cc:240] Negotiation successful
0926 21:36:45.330869 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":351,"threads_started":1}
I20260926 21:36:45.351219 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44101
I20260926 21:36:45.351240 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44101 every 8 connection(s)
I20260926 21:36:45.352586 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:36:45.355162 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42675
I20260926 21:36:45.355438 1857 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:45.355470 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.340307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38469 (local address 127.0.97.62:42675)
0926 21:36:45.340460 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:45.340465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:45.341212 (+ 747us) server_negotiation.cc:407] Connection header received
0926 21:36:45.341418 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:45.341427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:45.341505 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:45.341614 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:45.342944 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.343951 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.346010 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.346590 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.346726 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.349288 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:45.349294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:45.349297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:45.349327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:45.352312 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.353088 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.353091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.353094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.353310 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.353822 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.353827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.353830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.354296 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:45.354450 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:45.354924 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 21:36:45.355180 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:36:45.355857 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.340425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42675 (local address 127.0.97.2:38469)
0926 21:36:45.341077 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:45.341092 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:45.341285 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:45.341683 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:45.341688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:45.342349 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:45.342763 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.342777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.344094 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.344101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.345807 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.345825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.346790 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.346795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.346972 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.347790 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:45.347810 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:45.349048 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:45.352443 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:45.352451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:45.352457 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:45.352978 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:45.353445 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:45.353451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:45.353458 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:45.353647 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:45.354414 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:45.354418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:45.354640 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 21:36:45.354811 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 21:36:45.355172 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":410,"threads_started":1}
I20260926 21:36:45.356441 1857 heartbeater.cc:511] Master 127.0.97.62:42675 requested a full tablet report, sending...
I20260926 21:36:45.357461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 1722
I20260926 21:36:45.357622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:36:45.357811 1511 ts_manager.cc:194] Registered new tserver with Master: 39a77258e7ff44c9bff4db1c48176c6a (127.0.97.2:44101)
I20260926 21:36:45.358212 388 external_mini_cluster.cc:1428] Setting key aa0c4a4d832cc2342e37e24cee3530ac
I20260926 21:36:45.358732 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34139
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:36:45.398075 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42675
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:45.716102 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:45.716388 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:45.716483 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:45.726464 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:45.726632 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:45.726797 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:36:45.726940 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:36:45.738157 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:45.739512 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:45.740814 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:45.753471 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:45.753549 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:45.753549 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:45.754392 1861 server_base.cc:1034] running on GCE node
I20260926 21:36:45.754956 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:45.755861 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:45.757097 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790458605757030 us; error 45 us; skew 500 ppm
I20260926 21:36:45.757483 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:45 dist-test-slave-lhhq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:45.760393 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:36:45.761850 1861 webserver.cc:460] Webserver started at http://127.0.97.3:45799/ using document root <none> and password file <none>
I20260926 21:36:45.762523 1861 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:45.762650 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:45.762956 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:45.765802 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "fed6fea718054b5481d49a88831153b0"
format_stamp: "Formatted at 2026-09-26 21:36:45 on dist-test-slave-lhhq"
server_key: "cefad5d481af0542cf266b7c482b5fb3"
server_key_iv: "fec30f62d19680776fe89fe3aa656714"
server_key_version: "encryptionkey@0"
I20260926 21:36:45.766566 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fed6fea718054b5481d49a88831153b0"
format_stamp: "Formatted at 2026-09-26 21:36:45 on dist-test-slave-lhhq"
server_key: "cefad5d481af0542cf266b7c482b5fb3"
server_key_iv: "fec30f62d19680776fe89fe3aa656714"
server_key_version: "encryptionkey@0"
I20260926 21:36:45.771723 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:36:45.774996 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:45.776284 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 21:36:45.776458 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "fed6fea718054b5481d49a88831153b0"
format_stamp: "Formatted at 2026-09-26 21:36:45 on dist-test-slave-lhhq"
server_key: "cefad5d481af0542cf266b7c482b5fb3"
server_key_iv: "fec30f62d19680776fe89fe3aa656714"
server_key_version: "encryptionkey@0"
I20260926 21:36:45.776682 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:45.793337 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:45.797662 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:45.797942 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:45.800901 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:45.801020 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:45.801205 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:45.801302 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 26 21:36:45 dist-test-slave-lhhq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:45.825781 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.801675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52049 (local address 127.0.97.62:42675)
0926 21:36:45.801858 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:45.801862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:45.803552 (+ 1690us) server_negotiation.cc:407] Connection header received
0926 21:36:45.805843 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:45.805850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:45.805941 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:45.806061 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:45.807996 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.809122 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.811733 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.812536 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.812718 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.818138 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:45.818148 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:45.818152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:45.818188 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:45.821541 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.822458 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.822465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.822468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.822629 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.823263 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.823271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.823275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.823691 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:45.823854 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:45.824925 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0926 21:36:45.825282 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:36:45.827142 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.802477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42675 (local address 127.0.97.3:52049)
0926 21:36:45.803384 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:45.803424 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:45.805552 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:45.806393 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:45.806401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:45.806921 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:45.807749 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.807783 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.809328 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.809334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.811542 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.811566 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.812814 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.812825 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.813136 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.814052 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:45.814097 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:45.817828 (+ 3731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:45.821710 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:45.821720 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:45.821743 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:45.822229 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:45.822759 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:45.822767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:45.822771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:45.823150 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:45.823830 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:45.823839 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:45.824421 (+ 582us) client_negotiation.cc:769] Sending connection context
0926 21:36:45.824801 (+ 380us) client_negotiation.cc:240] Negotiation successful
0926 21:36:45.825476 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":412,"threads_started":1}
I20260926 21:36:45.849179 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.834326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37829 (local address 127.0.97.62:42675)
0926 21:36:45.834502 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:45.834508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:45.835304 (+ 796us) server_negotiation.cc:407] Connection header received
0926 21:36:45.835619 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:45.835625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:45.835751 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:45.835861 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:45.837379 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.838305 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.839603 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.840201 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.840355 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.842865 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:45.842873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:45.842881 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:45.842930 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:45.846407 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.847270 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.847276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.847279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.847411 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.847750 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.847757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.847761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.848026 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:45.848176 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:45.848584 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 21:36:45.848811 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:36:45.849304 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.834379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42675 (local address 127.0.97.3:37829)
0926 21:36:45.835173 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:45.835199 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:45.835472 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:45.835922 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:45.835929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:45.836720 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:45.837209 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.837224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.838435 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.838441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.839459 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.839470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.840428 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.840437 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.840718 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.841552 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:45.841569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:45.842662 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:45.846590 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:45.846598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:45.846604 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:45.847154 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:45.847519 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:45.847525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:45.847529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:45.847650 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:45.848141 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:45.848146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:45.848331 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 21:36:45.848481 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 21:36:45.848811 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":423,"threads_started":1}
I20260926 21:36:45.860802 1718 heartbeater.cc:503] Master 127.0.97.62:42675 was elected leader, sending a full tablet report...
I20260926 21:36:45.862408 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44869
I20260926 21:36:45.862429 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44869 every 8 connection(s)
I20260926 21:36:45.863600 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:36:45.865585 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42675
I20260926 21:36:45.865816 1996 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:45.866243 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 1861
I20260926 21:36:45.866407 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:36:45.866668 1996 heartbeater.cc:511] Master 127.0.97.62:42675 requested a full tablet report, sending...
I20260926 21:36:45.866987 388 external_mini_cluster.cc:1428] Setting key e4d0fffeab852f68e50c415662017599
I20260926 21:36:45.868372 1510 ts_manager.cc:194] Registered new tserver with Master: fed6fea718054b5481d49a88831153b0 (127.0.97.3:44869)
I20260926 21:36:45.869566 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52049
I20260926 21:36:45.872545 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:36:45 dist-test-slave-lhhq krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:36:45 dist-test-slave-lhhq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:45.917601 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.903622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.0.97.62:42675)
0926 21:36:45.903855 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:45.903863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:45.904130 (+ 267us) server_negotiation.cc:407] Connection header received
0926 21:36:45.904376 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:45.904381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:45.904459 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:45.904566 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:45.906013 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.906741 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.907994 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.908436 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.908566 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.912376 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:45.912381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:45.912384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:45.912418 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:45.914973 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.915610 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.915617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.915620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.915738 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:45.916115 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:45.916121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:45.916125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:45.916399 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:45.916630 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:45.916946 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 21:36:45.917175 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:36:45.921708 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:36:45.928395 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:36:45.960668 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.940297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33023 (local address 127.0.0.1:58416)
0926 21:36:45.941830 (+ 1533us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:45.941870 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:45.942041 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:45.949523 (+ 7482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:45.949531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:45.949563 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:45.950087 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.950104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.952887 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.952896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.956195 (+ 3299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.956216 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.959010 (+ 2794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.959020 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.959482 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.959605 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 21:36:45.959784 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 21:36:45.960053 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1251,"mutex_wait_us":74,"thread_start_us":8506,"threads_started":1}
I20260926 21:36:45.962191 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.940854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.0.97.1:33023)
0926 21:36:45.948866 (+ 8012us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:45.948872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:45.948898 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:36:45.948974 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:45.948979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:45.949253 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:45.950384 (+ 1131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:45.950637 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.952694 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.956525 (+ 3831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.958689 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.961252 (+ 2563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.961590 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:45.961672 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 21:36:45.961853 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7866,"thread_start_us":7737,"threads_started":1}
I20260926 21:36:45.963948 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.940318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44869 (local address 127.0.0.1:60870)
0926 21:36:45.949075 (+ 8757us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:45.949093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:45.949249 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:45.950395 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:45.950403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:45.950439 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:45.950937 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.952193 (+ 1256us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.954849 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.954858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.959559 (+ 4701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.959580 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.963137 (+ 3557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.963142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.963345 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.963378 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:36:45.963456 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 21:36:45.963593 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8632,"mutex_wait_us":118,"thread_start_us":1072,"threads_started":1}
I20260926 21:36:45.964294 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.940388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.0.97.3:44869)
0926 21:36:45.948585 (+ 8197us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:45.948598 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:45.949391 (+ 793us) server_negotiation.cc:407] Connection header received
0926 21:36:45.949510 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:45.949521 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:45.949824 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:45.950043 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:45.952447 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.954632 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.959824 (+ 5192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.961904 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.962068 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.963721 (+ 1653us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:45.963808 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 21:36:45.963969 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7963,"thread_start_us":7715,"threads_started":1}
I20260926 21:36:45.966868 1648 tablet_service.cc:1467] Processing CreateTablet for tablet a663d84e9eea455b85eb173544bfe7e4 (DEFAULT_TABLE table=test-table [id=1e8604cc3c2040ab9d40fadd58a223c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:45.967545 1926 tablet_service.cc:1467] Processing CreateTablet for tablet a663d84e9eea455b85eb173544bfe7e4 (DEFAULT_TABLE table=test-table [id=1e8604cc3c2040ab9d40fadd58a223c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:45.968381 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a663d84e9eea455b85eb173544bfe7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:45.968878 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a663d84e9eea455b85eb173544bfe7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:45.968881 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.949271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44101 (local address 127.0.0.1:60576)
0926 21:36:45.960925 (+ 11654us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:45.960942 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:45.961051 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:45.962881 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:45.962892 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:45.962923 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:45.963138 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.963148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.965799 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.965806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.966917 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:45.966927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.968156 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.968162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:45.968360 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.968395 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:36:45.968480 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 21:36:45.968613 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11559,"thread_start_us":13551,"threads_started":1}
I20260926 21:36:45.969158 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:45.949493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.0.97.2:44101)
0926 21:36:45.961993 (+ 12500us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:45.962003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:45.962036 (+ 33us) server_negotiation.cc:407] Connection header received
0926 21:36:45.962136 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:45.962144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:45.962610 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:45.963318 (+ 708us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:45.963503 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.965657 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.967069 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:45.967984 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:45.968122 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:45.968309 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:45.968537 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 21:36:45.968690 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12329,"thread_start_us":12962,"threads_started":1}
I20260926 21:36:45.972076 1787 tablet_service.cc:1467] Processing CreateTablet for tablet a663d84e9eea455b85eb173544bfe7e4 (DEFAULT_TABLE table=test-table [id=1e8604cc3c2040ab9d40fadd58a223c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:45.973170 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a663d84e9eea455b85eb173544bfe7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:45.984313 2015 tablet_bootstrap.cc:492] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: Bootstrap starting.
I20260926 21:36:45.986447 2016 tablet_bootstrap.cc:492] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: Bootstrap starting.
I20260926 21:36:45.988130 2017 tablet_bootstrap.cc:492] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: Bootstrap starting.
I20260926 21:36:45.988529 2015 tablet_bootstrap.cc:654] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:45.990020 2015 log.cc:824] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:45.990239 2016 tablet_bootstrap.cc:654] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:45.991369 2016 log.cc:824] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:45.991454 2017 tablet_bootstrap.cc:654] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:45.992849 2017 log.cc:824] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:45.993917 2015 tablet_bootstrap.cc:492] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: No bootstrap required, opened a new log
I20260926 21:36:45.994207 2015 ts_tablet_manager.cc:1397] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 21:36:45.994580 2016 tablet_bootstrap.cc:492] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: No bootstrap required, opened a new log
I20260926 21:36:45.994840 2016 ts_tablet_manager.cc:1397] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 21:36:45.995640 2017 tablet_bootstrap.cc:492] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: No bootstrap required, opened a new log
I20260926 21:36:45.995901 2017 ts_tablet_manager.cc:1397] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 21:36:46.002249 2016 raft_consensus.cc:348] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } }
I20260926 21:36:46.002657 2016 raft_consensus.cc:374] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:46.002759 2016 raft_consensus.cc:729] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed6fea718054b5481d49a88831153b0, State: Initialized, Role: FOLLOWER
I20260926 21:36:46.003485 2016 consensus_queue.cc:260] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } }
I20260926 21:36:46.004853 1996 heartbeater.cc:503] Master 127.0.97.62:42675 was elected leader, sending a full tablet report...
I20260926 21:36:46.005605 2016 ts_tablet_manager.cc:1428] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260926 21:36:46.005502 2015 raft_consensus.cc:348] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } }
I20260926 21:36:46.006348 2015 raft_consensus.cc:374] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:46.006461 2015 raft_consensus.cc:729] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a77258e7ff44c9bff4db1c48176c6a, State: Initialized, Role: FOLLOWER
I20260926 21:36:46.007077 2015 consensus_queue.cc:260] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } }
I20260926 21:36:46.007061 2017 raft_consensus.cc:348] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } }
I20260926 21:36:46.007449 2017 raft_consensus.cc:374] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:46.007552 2017 raft_consensus.cc:729] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc9fa43b4045a28c208cacab6e8bbd, State: Initialized, Role: FOLLOWER
I20260926 21:36:46.008294 1857 heartbeater.cc:503] Master 127.0.97.62:42675 was elected leader, sending a full tablet report...
I20260926 21:36:46.008793 2015 ts_tablet_manager.cc:1428] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 21:36:46.009298 2017 consensus_queue.cc:260] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } }
I20260926 21:36:46.011266 2017 ts_tablet_manager.cc:1428] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260926 21:36:46.103596 1719 tablet.cc:2369] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:36:46.104700 1858 tablet.cc:2369] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:36:46.115136 1997 tablet.cc:2369] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:36:46.219841 2022 raft_consensus.cc:482] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:36:46.220153 2022 raft_consensus.cc:504] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } }
I20260926 21:36:46.221954 2022 leader_election.cc:288] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed6fea718054b5481d49a88831153b0 (127.0.97.3:44869), d7cc9fa43b4045a28c208cacab6e8bbd (127.0.97.1:33023)
I20260926 21:36:46.229321 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.222525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59389 (local address 127.0.97.3:44869)
0926 21:36:46.222741 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.222748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.223556 (+ 808us) server_negotiation.cc:407] Connection header received
0926 21:36:46.223737 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.223741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.223808 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.223914 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:46.224560 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.225801 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.227542 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.228289 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.228417 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.228537 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.228855 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.229021 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:36:46.229337 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.222388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37547 (local address 127.0.97.1:33023)
0926 21:36:46.222594 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.222602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.223427 (+ 825us) server_negotiation.cc:407] Connection header received
0926 21:36:46.223699 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.223706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.223795 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.223866 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:46.224586 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.225633 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.227572 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.228301 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.228460 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.228566 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.228943 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.229066 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:36:46.229528 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.222458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44869 (local address 127.0.97.2:59389)
0926 21:36:46.223444 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:46.223458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:46.223574 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:46.223943 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:46.223949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:46.223969 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:46.224402 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.224412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.225932 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.225935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.227393 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.227404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.228517 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.228524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.228687 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.228754 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 21:36:46.228824 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 21:36:46.229022 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":44,"thread_start_us":657,"threads_started":1}
I20260926 21:36:46.229552 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.222389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33023 (local address 127.0.97.2:37547)
0926 21:36:46.223306 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:46.223323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:46.223500 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:46.223914 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:46.223919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:46.223941 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:46.224418 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.224427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.225760 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.225765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.227397 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.227407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.228548 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.228557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.228699 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.228801 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 21:36:46.228895 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 21:36:46.229056 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":153,"thread_start_us":559,"threads_started":1}
I20260926 21:36:46.230605 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663d84e9eea455b85eb173544bfe7e4" candidate_uuid: "39a77258e7ff44c9bff4db1c48176c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed6fea718054b5481d49a88831153b0" is_pre_election: true
I20260926 21:36:46.230772 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663d84e9eea455b85eb173544bfe7e4" candidate_uuid: "39a77258e7ff44c9bff4db1c48176c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" is_pre_election: true
I20260926 21:36:46.231141 1946 raft_consensus.cc:2454] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a77258e7ff44c9bff4db1c48176c6a in term 0.
I20260926 21:36:46.231233 1668 raft_consensus.cc:2454] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a77258e7ff44c9bff4db1c48176c6a in term 0.
I20260926 21:36:46.231839 1738 leader_election.cc:302] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a77258e7ff44c9bff4db1c48176c6a, d7cc9fa43b4045a28c208cacab6e8bbd; no voters:
I20260926 21:36:46.232249 2022 raft_consensus.cc:2781] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:36:46.232425 2022 raft_consensus.cc:482] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:36:46.232501 2022 raft_consensus.cc:3037] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:46.235188 2022 raft_consensus.cc:504] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } }
I20260926 21:36:46.235771 2022 leader_election.cc:288] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [CANDIDATE]: Term 1 election: Requested vote from peers fed6fea718054b5481d49a88831153b0 (127.0.97.3:44869), d7cc9fa43b4045a28c208cacab6e8bbd (127.0.97.1:33023)
I20260926 21:36:46.236197 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663d84e9eea455b85eb173544bfe7e4" candidate_uuid: "39a77258e7ff44c9bff4db1c48176c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed6fea718054b5481d49a88831153b0"
I20260926 21:36:46.236264 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663d84e9eea455b85eb173544bfe7e4" candidate_uuid: "39a77258e7ff44c9bff4db1c48176c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd"
I20260926 21:36:46.236392 1946 raft_consensus.cc:3037] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:46.236478 1668 raft_consensus.cc:3037] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:46.239230 1668 raft_consensus.cc:2454] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a77258e7ff44c9bff4db1c48176c6a in term 1.
I20260926 21:36:46.239645 1738 leader_election.cc:302] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a77258e7ff44c9bff4db1c48176c6a, d7cc9fa43b4045a28c208cacab6e8bbd; no voters:
I20260926 21:36:46.239929 2022 raft_consensus.cc:2781] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:46.239957 1946 raft_consensus.cc:2454] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a77258e7ff44c9bff4db1c48176c6a in term 1.
I20260926 21:36:46.240638 2022 raft_consensus.cc:686] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 1 LEADER]: Becoming Leader. State: Replica: 39a77258e7ff44c9bff4db1c48176c6a, State: Running, Role: LEADER
I20260926 21:36:46.241227 2022 consensus_queue.cc:237] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } }
I20260926 21:36:46.245744 1510 catalog_manager.cc:5495] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 39a77258e7ff44c9bff4db1c48176c6a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "39a77258e7ff44c9bff4db1c48176c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } health_report { overall_health: HEALTHY } } }
I20260926 21:36:46.322845 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.317291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.0.97.2:44101)
0926 21:36:46.317493 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.317500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.317638 (+ 138us) server_negotiation.cc:407] Connection header received
0926 21:36:46.317976 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.317980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.318061 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.318206 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:46.318583 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.319329 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.320656 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.321094 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.321246 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.321735 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:46.322252 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:46.322403 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.322491 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.322570 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:36:46.331046 1668 raft_consensus.cc:1261] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 FOLLOWER]: Refusing update from remote peer 39a77258e7ff44c9bff4db1c48176c6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:46.331067 1946 raft_consensus.cc:1261] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Refusing update from remote peer 39a77258e7ff44c9bff4db1c48176c6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:46.332069 2022 consensus_queue.cc:1035] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:46.332387 2026 consensus_queue.cc:1035] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:46.345207 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333718451521806336 to 7333718451168149504. Current Snapshot: MvccSnapshot[applied={T|T < 7333718451521806336}]
I20260926 21:36:46.346032 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333718451521806336 to 7333718451168149504. Current Snapshot: MvccSnapshot[applied={T|T < 7333718451521806336}]
I20260926 21:36:46.370417 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:46.370769 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:46.374653 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f: Sending DeleteTablet for 3 replicas of tablet a663d84e9eea455b85eb173544bfe7e4
I20260926 21:36:46.375888 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet a663d84e9eea455b85eb173544bfe7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60870
I20260926 21:36:46.376240 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet a663d84e9eea455b85eb173544bfe7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58416
I20260926 21:36:46.376593 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet a663d84e9eea455b85eb173544bfe7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60576
I20260926 21:36:46.377429 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:36:46.378338 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:36:46.388386 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 63a5dfe8b6a648e590618f51cf537bba (DEFAULT_TABLE table=test-table [id=4689c9aca7d445a497a008b0ce941420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:46.388576 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 63a5dfe8b6a648e590618f51cf537bba (DEFAULT_TABLE table=test-table [id=4689c9aca7d445a497a008b0ce941420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:46.388967 2041 tablet_replica.cc:330] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: stopping tablet replica
I20260926 21:36:46.389036 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a5dfe8b6a648e590618f51cf537bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:46.389498 2041 raft_consensus.cc:2229] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:36:46.389662 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a5dfe8b6a648e590618f51cf537bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:46.391403 2041 raft_consensus.cc:2258] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:46.392756 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 63a5dfe8b6a648e590618f51cf537bba (DEFAULT_TABLE table=test-table [id=4689c9aca7d445a497a008b0ce941420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:46.393313 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a5dfe8b6a648e590618f51cf537bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:46.396422 2017 tablet_bootstrap.cc:492] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: Bootstrap starting.
I20260926 21:36:46.396607 2016 tablet_bootstrap.cc:492] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: Bootstrap starting.
I20260926 21:36:46.398233 2017 tablet_bootstrap.cc:654] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:46.398747 2016 tablet_bootstrap.cc:654] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:46.404115 2015 tablet_bootstrap.cc:492] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: Bootstrap starting.
I20260926 21:36:46.405771 2016 tablet_bootstrap.cc:492] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: No bootstrap required, opened a new log
I20260926 21:36:46.406008 2016 ts_tablet_manager.cc:1397] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260926 21:36:46.406275 2015 tablet_bootstrap.cc:654] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:46.406935 2016 raft_consensus.cc:348] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.407235 2016 raft_consensus.cc:374] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:46.407390 2016 raft_consensus.cc:729] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed6fea718054b5481d49a88831153b0, State: Initialized, Role: FOLLOWER
I20260926 21:36:46.407729 2016 consensus_queue.cc:260] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.410924 2016 ts_tablet_manager.cc:1428] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 21:36:46.414577 2041 ts_tablet_manager.cc:1905] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:46.415361 2017 tablet_bootstrap.cc:492] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: No bootstrap required, opened a new log
I20260926 21:36:46.415562 2017 ts_tablet_manager.cc:1397] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20260926 21:36:46.416675 2017 raft_consensus.cc:348] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.416997 2017 raft_consensus.cc:374] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:46.417114 2017 raft_consensus.cc:729] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc9fa43b4045a28c208cacab6e8bbd, State: Initialized, Role: FOLLOWER
I20260926 21:36:46.417484 2017 consensus_queue.cc:260] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.419018 2017 ts_tablet_manager.cc:1428] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 21:36:46.422822 2040 tablet_replica.cc:330] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: stopping tablet replica
I20260926 21:36:46.423354 2040 raft_consensus.cc:2229] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:46.423870 2040 raft_consensus.cc:2258] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:46.424441 2039 tablet_replica.cc:330] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: stopping tablet replica
I20260926 21:36:46.425026 2039 raft_consensus.cc:2229] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:46.426011 2039 raft_consensus.cc:2258] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:46.428596 2039 ts_tablet_manager.cc:1905] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:46.429916 2041 ts_tablet_manager.cc:1918] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:36:46.430155 2041 log.cc:1196] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/a663d84e9eea455b85eb173544bfe7e4
I20260926 21:36:46.430632 2015 tablet_bootstrap.cc:492] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: No bootstrap required, opened a new log
I20260926 21:36:46.430814 2015 ts_tablet_manager.cc:1397] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.008s
I20260926 21:36:46.431072 2041 ts_tablet_manager.cc:1939] T a663d84e9eea455b85eb173544bfe7e4 P 39a77258e7ff44c9bff4db1c48176c6a: Deleting consensus metadata
I20260926 21:36:46.431687 2015 raft_consensus.cc:348] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.431908 2015 raft_consensus.cc:374] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:46.432009 2015 raft_consensus.cc:729] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a77258e7ff44c9bff4db1c48176c6a, State: Initialized, Role: FOLLOWER
I20260926 21:36:46.432343 2015 consensus_queue.cc:260] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.432838 2023 raft_consensus.cc:482] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:36:46.433022 2023 raft_consensus.cc:504] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.433707 1495 catalog_manager.cc:4841] TS 39a77258e7ff44c9bff4db1c48176c6a (127.0.97.2:44101): tablet a663d84e9eea455b85eb173544bfe7e4 (table test-table [id=1e8604cc3c2040ab9d40fadd58a223c4]) successfully deleted
I20260926 21:36:46.435230 2023 leader_election.cc:288] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39a77258e7ff44c9bff4db1c48176c6a (127.0.97.2:44101), fed6fea718054b5481d49a88831153b0 (127.0.97.3:44869)
I20260926 21:36:46.436314 2039 ts_tablet_manager.cc:1918] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:36:46.436487 2039 log.cc:1196] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/a663d84e9eea455b85eb173544bfe7e4
I20260926 21:36:46.437039 2039 ts_tablet_manager.cc:1939] T a663d84e9eea455b85eb173544bfe7e4 P fed6fea718054b5481d49a88831153b0: Deleting consensus metadata
I20260926 21:36:46.439013 1498 catalog_manager.cc:4841] TS fed6fea718054b5481d49a88831153b0 (127.0.97.3:44869): tablet a663d84e9eea455b85eb173544bfe7e4 (table test-table [id=1e8604cc3c2040ab9d40fadd58a223c4]) successfully deleted
I20260926 21:36:46.441745 2015 ts_tablet_manager.cc:1428] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20260926 21:36:46.441855 2040 ts_tablet_manager.cc:1905] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:46.450323 2040 ts_tablet_manager.cc:1918] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:36:46.450520 2040 log.cc:1196] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/a663d84e9eea455b85eb173544bfe7e4
I20260926 21:36:46.451056 2040 ts_tablet_manager.cc:1939] T a663d84e9eea455b85eb173544bfe7e4 P d7cc9fa43b4045a28c208cacab6e8bbd: Deleting consensus metadata
I20260926 21:36:46.451512 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.437317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57577 (local address 127.0.97.3:44869)
0926 21:36:46.437489 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.437495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.443227 (+ 5732us) server_negotiation.cc:407] Connection header received
0926 21:36:46.443482 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.443489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.443570 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.443629 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:46.444495 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.445607 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.448772 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.449985 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.450089 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.450234 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.451084 (+ 850us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.451239 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:36:46.452409 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.436785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44869 (local address 127.0.97.1:57577)
0926 21:36:46.443117 (+ 6332us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:46.443135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:46.443324 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:46.443818 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:46.443822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:46.443849 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:46.444274 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.444289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.445794 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.445798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.448589 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.448601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.450551 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.450560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.450886 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.450922 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 21:36:46.451858 (+ 936us) client_negotiation.cc:240] Negotiation successful
0926 21:36:46.452004 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6177,"thread_start_us":8093,"threads_started":1}
I20260926 21:36:46.452942 1495 catalog_manager.cc:4841] TS d7cc9fa43b4045a28c208cacab6e8bbd (127.0.97.1:33023): tablet a663d84e9eea455b85eb173544bfe7e4 (table test-table [id=1e8604cc3c2040ab9d40fadd58a223c4]) successfully deleted
I20260926 21:36:46.453526 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.437603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56691 (local address 127.0.97.2:44101)
0926 21:36:46.437770 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.437776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.442649 (+ 4873us) server_negotiation.cc:407] Connection header received
0926 21:36:46.442915 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.442924 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.443001 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.443101 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:46.446488 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.448113 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.450694 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.451960 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.452069 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.452176 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.453016 (+ 840us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.453175 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:36:46.453821 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.436260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44101 (local address 127.0.97.1:56691)
0926 21:36:46.442469 (+ 6209us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:46.442498 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:46.442650 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:46.445225 (+ 2575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:46.445229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:46.445253 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:46.445585 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.445594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.448308 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.448313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.450512 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.450524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.452468 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.452475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.452684 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.452853 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 21:36:46.452936 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 21:36:46.453069 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6071,"thread_start_us":8773,"threads_started":1}
I20260926 21:36:46.453732 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a5dfe8b6a648e590618f51cf537bba" candidate_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed6fea718054b5481d49a88831153b0" is_pre_election: true
I20260926 21:36:46.453995 1946 raft_consensus.cc:2454] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7cc9fa43b4045a28c208cacab6e8bbd in term 0.
I20260926 21:36:46.455057 1601 leader_election.cc:302] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7cc9fa43b4045a28c208cacab6e8bbd, fed6fea718054b5481d49a88831153b0; no voters:
I20260926 21:36:46.455193 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a5dfe8b6a648e590618f51cf537bba" candidate_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a77258e7ff44c9bff4db1c48176c6a" is_pre_election: true
I20260926 21:36:46.455510 2023 raft_consensus.cc:2781] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:36:46.455632 2023 raft_consensus.cc:482] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:36:46.455631 1807 raft_consensus.cc:2454] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7cc9fa43b4045a28c208cacab6e8bbd in term 0.
I20260926 21:36:46.455754 2023 raft_consensus.cc:3037] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:46.458140 2023 raft_consensus.cc:504] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.459417 2023 leader_election.cc:288] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [CANDIDATE]: Term 1 election: Requested vote from peers 39a77258e7ff44c9bff4db1c48176c6a (127.0.97.2:44101), fed6fea718054b5481d49a88831153b0 (127.0.97.3:44869)
I20260926 21:36:46.459363 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a5dfe8b6a648e590618f51cf537bba" candidate_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a77258e7ff44c9bff4db1c48176c6a"
I20260926 21:36:46.459594 1807 raft_consensus.cc:3037] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:46.460172 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a5dfe8b6a648e590618f51cf537bba" candidate_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed6fea718054b5481d49a88831153b0"
I20260926 21:36:46.460397 1946 raft_consensus.cc:3037] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:46.462184 1807 raft_consensus.cc:2454] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc9fa43b4045a28c208cacab6e8bbd in term 1.
I20260926 21:36:46.462658 1598 leader_election.cc:302] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a77258e7ff44c9bff4db1c48176c6a, d7cc9fa43b4045a28c208cacab6e8bbd; no voters:
I20260926 21:36:46.462960 2023 raft_consensus.cc:2781] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:46.463442 1946 raft_consensus.cc:2454] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc9fa43b4045a28c208cacab6e8bbd in term 1.
I20260926 21:36:46.463516 2023 raft_consensus.cc:686] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 LEADER]: Becoming Leader. State: Replica: d7cc9fa43b4045a28c208cacab6e8bbd, State: Running, Role: LEADER
I20260926 21:36:46.464186 2023 consensus_queue.cc:237] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.468195 1511 catalog_manager.cc:5495] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd reported cstate change: term changed from 0 to 1, leader changed from <none> to d7cc9fa43b4045a28c208cacab6e8bbd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
Sep 26 21:36:46 dist-test-slave-lhhq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:46.529919 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.515752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42675 (local address 127.0.0.1:34948)
0926 21:36:46.516573 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:46.516588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:46.516764 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:46.517100 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:46.517104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:46.517837 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:46.518081 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.518090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.519182 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.519189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.520156 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.520167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.520895 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.520901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.521092 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.521884 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:46.521902 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:46.524491 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:46.527491 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:46.527506 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:46.527514 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:46.527917 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:46.528265 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:46.528270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:46.528272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:46.528421 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:46.528954 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:46.528963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:46.529161 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 21:36:46.529418 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 21:36:46.529624 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":554,"threads_started":1}
I20260926 21:36:46.530711 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.515689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.0.97.62:42675)
0926 21:36:46.516382 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.516390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.516682 (+ 292us) server_negotiation.cc:407] Connection header received
0926 21:36:46.516905 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.516908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.516966 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.517074 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:46.518244 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.519052 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.520333 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.520724 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.520874 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.524724 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:46.524729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:46.524732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:46.524761 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:46.527358 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:46.528033 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:46.528041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:46.528044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:46.528163 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:46.528529 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:46.528533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:46.528537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:46.528836 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:46.528979 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.530237 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.530382 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":375,"threads_started":1}
I20260926 21:36:46.535975 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34948:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 21:36:46.536994 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:36:46.544657 1648 tablet_service.cc:1467] Processing CreateTablet for tablet f06407e9fb0a4c34b649a4217cae16c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4635b04c132441da0712b66d575eb1d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:36:46.545020 1926 tablet_service.cc:1467] Processing CreateTablet for tablet f06407e9fb0a4c34b649a4217cae16c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4635b04c132441da0712b66d575eb1d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:36:46.545327 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06407e9fb0a4c34b649a4217cae16c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:46.545600 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06407e9fb0a4c34b649a4217cae16c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:46.550238 1786 tablet_service.cc:1467] Processing CreateTablet for tablet f06407e9fb0a4c34b649a4217cae16c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4635b04c132441da0712b66d575eb1d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:36:46.550537 2017 tablet_bootstrap.cc:492] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: Bootstrap starting.
I20260926 21:36:46.550789 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f06407e9fb0a4c34b649a4217cae16c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:46.552675 2017 tablet_bootstrap.cc:654] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:46.552996 2016 tablet_bootstrap.cc:492] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: Bootstrap starting.
I20260926 21:36:46.554687 2016 tablet_bootstrap.cc:654] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:46.561579 2015 tablet_bootstrap.cc:492] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: Bootstrap starting.
I20260926 21:36:46.564292 2015 tablet_bootstrap.cc:654] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:46.565259 2017 tablet_bootstrap.cc:492] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: No bootstrap required, opened a new log
I20260926 21:36:46.565550 2017 ts_tablet_manager.cc:1397] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260926 21:36:46.566610 2017 raft_consensus.cc:348] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.567024 2017 raft_consensus.cc:374] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:46.567174 2017 raft_consensus.cc:729] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc9fa43b4045a28c208cacab6e8bbd, State: Initialized, Role: FOLLOWER
I20260926 21:36:46.567584 2017 consensus_queue.cc:260] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.568223 2023 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.568503 2023 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:46.569222 2017 ts_tablet_manager.cc:1428] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 21:36:46.570660 2016 tablet_bootstrap.cc:492] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: No bootstrap required, opened a new log
I20260926 21:36:46.570833 2016 ts_tablet_manager.cc:1397] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.001s
I20260926 21:36:46.571643 2016 raft_consensus.cc:348] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.571815 2015 tablet_bootstrap.cc:492] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: No bootstrap required, opened a new log
I20260926 21:36:46.571892 2016 raft_consensus.cc:374] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:46.571972 2016 raft_consensus.cc:729] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed6fea718054b5481d49a88831153b0, State: Initialized, Role: FOLLOWER
I20260926 21:36:46.571990 2015 ts_tablet_manager.cc:1397] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 21:36:46.572227 2016 consensus_queue.cc:260] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.572744 2016 ts_tablet_manager.cc:1428] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 21:36:46.572816 2015 raft_consensus.cc:348] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.573076 2015 raft_consensus.cc:374] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:46.573167 2015 raft_consensus.cc:729] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a77258e7ff44c9bff4db1c48176c6a, State: Initialized, Role: FOLLOWER
I20260926 21:36:46.573060 2021 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.573431 2021 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:46.573490 2015 consensus_queue.cc:260] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.574445 2034 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.574980 2034 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:46.576009 2015 ts_tablet_manager.cc:1428] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 21:36:46.673532 2034 raft_consensus.cc:482] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:36:46.673751 2034 raft_consensus.cc:504] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.674584 2034 leader_election.cc:288] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cc9fa43b4045a28c208cacab6e8bbd (127.0.97.1:33023), fed6fea718054b5481d49a88831153b0 (127.0.97.3:44869)
I20260926 21:36:46.675140 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06407e9fb0a4c34b649a4217cae16c1" candidate_uuid: "39a77258e7ff44c9bff4db1c48176c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed6fea718054b5481d49a88831153b0" is_pre_election: true
I20260926 21:36:46.675194 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06407e9fb0a4c34b649a4217cae16c1" candidate_uuid: "39a77258e7ff44c9bff4db1c48176c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" is_pre_election: true
I20260926 21:36:46.675415 1946 raft_consensus.cc:2454] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a77258e7ff44c9bff4db1c48176c6a in term 0.
I20260926 21:36:46.675478 1668 raft_consensus.cc:2454] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a77258e7ff44c9bff4db1c48176c6a in term 0.
I20260926 21:36:46.675910 1741 leader_election.cc:302] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a77258e7ff44c9bff4db1c48176c6a, fed6fea718054b5481d49a88831153b0; no voters:
I20260926 21:36:46.676277 2034 raft_consensus.cc:2781] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:36:46.676458 2034 raft_consensus.cc:482] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:36:46.676558 2034 raft_consensus.cc:3037] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:46.679286 2034 raft_consensus.cc:504] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.679932 2034 leader_election.cc:288] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [CANDIDATE]: Term 1 election: Requested vote from peers d7cc9fa43b4045a28c208cacab6e8bbd (127.0.97.1:33023), fed6fea718054b5481d49a88831153b0 (127.0.97.3:44869)
I20260926 21:36:46.680513 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06407e9fb0a4c34b649a4217cae16c1" candidate_uuid: "39a77258e7ff44c9bff4db1c48176c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed6fea718054b5481d49a88831153b0"
I20260926 21:36:46.680722 1946 raft_consensus.cc:3037] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:46.682747 1946 raft_consensus.cc:2454] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a77258e7ff44c9bff4db1c48176c6a in term 1.
I20260926 21:36:46.683179 1741 leader_election.cc:302] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a77258e7ff44c9bff4db1c48176c6a, fed6fea718054b5481d49a88831153b0; no voters:
I20260926 21:36:46.683185 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06407e9fb0a4c34b649a4217cae16c1" candidate_uuid: "39a77258e7ff44c9bff4db1c48176c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd"
I20260926 21:36:46.683395 1668 raft_consensus.cc:3037] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:46.683538 2034 raft_consensus.cc:2781] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:46.683885 2034 raft_consensus.cc:686] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [term 1 LEADER]: Becoming Leader. State: Replica: 39a77258e7ff44c9bff4db1c48176c6a, State: Running, Role: LEADER
I20260926 21:36:46.684166 2034 consensus_queue.cc:237] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } }
I20260926 21:36:46.686914 1668 raft_consensus.cc:2454] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a77258e7ff44c9bff4db1c48176c6a in term 1.
I20260926 21:36:46.688045 2026 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: TxnStatusTablet state changed. Reason: New leader 39a77258e7ff44c9bff4db1c48176c6a. Latest consensus state: current_term: 1 leader_uuid: "39a77258e7ff44c9bff4db1c48176c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.688419 2026 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:36:46.689666 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 21:36:46.689913 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 21:36:46.691108 1509 catalog_manager.cc:5495] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 39a77258e7ff44c9bff4db1c48176c6a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "39a77258e7ff44c9bff4db1c48176c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } health_report { overall_health: UNKNOWN } } }
I20260926 21:36:46.730079 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.724278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.97.2:44101)
0926 21:36:46.724457 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.724465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.724620 (+ 155us) server_negotiation.cc:407] Connection header received
0926 21:36:46.724897 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.724901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.724983 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.725064 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:46.725495 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.726227 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.728128 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.728589 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.728682 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.729279 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:46.729390 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:46.729463 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.729647 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.729710 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:36:46.730131 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.724238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44101 (local address 127.0.0.1:60590)
0926 21:36:46.724519 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:46.724541 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:46.724708 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:46.725125 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:46.725132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:46.725145 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:36:46.725363 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.725376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.726387 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.726392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.727982 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.727999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.728741 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.728747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.728985 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.729060 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:36:46.729494 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:36:46.729534 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 21:36:46.729631 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 21:36:46.729811 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 21:36:46.733891 1946 raft_consensus.cc:1261] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Refusing update from remote peer 39a77258e7ff44c9bff4db1c48176c6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:46.734017 1668 raft_consensus.cc:1261] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 FOLLOWER]: Refusing update from remote peer 39a77258e7ff44c9bff4db1c48176c6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:46.734741 2026 consensus_queue.cc:1035] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:46.735069 2034 consensus_queue.cc:1035] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:46.737816 2023 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: TxnStatusTablet state changed. Reason: New leader 39a77258e7ff44c9bff4db1c48176c6a. Latest consensus state: current_term: 1 leader_uuid: "39a77258e7ff44c9bff4db1c48176c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.738092 2023 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:46.740178 2034 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39a77258e7ff44c9bff4db1c48176c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.740469 2034 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:36:46.742740 2023 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39a77258e7ff44c9bff4db1c48176c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.743040 2023 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P d7cc9fa43b4045a28c208cacab6e8bbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:46.746060 2021 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: TxnStatusTablet state changed. Reason: New leader 39a77258e7ff44c9bff4db1c48176c6a. Latest consensus state: current_term: 1 leader_uuid: "39a77258e7ff44c9bff4db1c48176c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.746367 2021 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:46.748119 2022 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39a77258e7ff44c9bff4db1c48176c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.748428 2022 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P 39a77258e7ff44c9bff4db1c48176c6a: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:36:46.751391 2021 tablet_replica.cc:439] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39a77258e7ff44c9bff4db1c48176c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc9fa43b4045a28c208cacab6e8bbd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33023 } } peers { permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 } } peers { permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 } } }
I20260926 21:36:46.751631 2021 tablet_replica.cc:442] T f06407e9fb0a4c34b649a4217cae16c1 P fed6fea718054b5481d49a88831153b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:46.757663 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.749364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.0.97.1:33023)
0926 21:36:46.750485 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.750491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.750517 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:36:46.750615 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.751163 (+ 548us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.751290 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.752133 (+ 843us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:46.752292 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.753452 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.754853 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.755536 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.755665 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.756160 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:46.756948 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:46.757125 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.757179 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.757303 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":797,"threads_started":1}
I20260926 21:36:46.775177 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.768948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44101 (local address 127.0.97.1:42175)
0926 21:36:46.769640 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:46.769656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:46.769776 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:46.770127 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:46.770130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:46.770139 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:36:46.770479 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.770496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.772008 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.772012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.773102 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.773113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.774135 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.774139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.774260 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.774316 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:36:46.774722 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:36:46.774730 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 21:36:46.774789 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 21:36:46.774894 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":377,"threads_started":1}
I20260926 21:36:46.775233 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.769431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42175 (local address 127.0.97.2:44101)
0926 21:36:46.769626 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.769633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.769739 (+ 106us) server_negotiation.cc:407] Connection header received
0926 21:36:46.769907 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.769912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.770007 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.770112 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:46.770648 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.771869 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.773282 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.773969 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.774100 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.774495 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:46.774622 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:46.774718 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.774838 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.774921 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:36:46.786072 1807 raft_consensus.cc:1261] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [term 1 FOLLOWER]: Refusing update from remote peer d7cc9fa43b4045a28c208cacab6e8bbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:46.786926 1945 raft_consensus.cc:1261] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Refusing update from remote peer d7cc9fa43b4045a28c208cacab6e8bbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:46.786957 2047 consensus_queue.cc:1035] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a77258e7ff44c9bff4db1c48176c6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:46.787518 2023 consensus_queue.cc:1035] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6fea718054b5481d49a88831153b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:46.805091 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333718453387427840 to 7333718452082077696. Current Snapshot: MvccSnapshot[applied={T|T < 7333718453387427840}]
I20260926 21:36:46.827008 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.820103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36367 (local address 127.0.97.1:33023)
0926 21:36:46.820278 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.820286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.820933 (+ 647us) server_negotiation.cc:407] Connection header received
0926 21:36:46.821245 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.821250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.821357 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.821451 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:46.822125 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.822960 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.824541 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.825072 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.825224 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.826099 (+ 875us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:46.826237 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:46.826337 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.826586 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.826672 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:36:46.827508 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.820077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33023 (local address 127.0.97.2:36367)
0926 21:36:46.820828 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:46.820850 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:46.821058 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:46.821503 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:46.821508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:46.821524 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:36:46.821933 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.821953 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.823204 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.823214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.824404 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:46.824415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.825661 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.825668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:46.825822 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.825901 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:36:46.826403 (+ 502us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:36:46.826418 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 21:36:46.826555 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 21:36:46.826719 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":421,"threads_started":1}
I20260926 21:36:46.872258 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:46.872545 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:46.876160 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4392a8865234599a9725e8a35d9367f: Sending DeleteTablet for 3 replicas of tablet 63a5dfe8b6a648e590618f51cf537bba
I20260926 21:36:46.877646 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a5dfe8b6a648e590618f51cf537bba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58416
I20260926 21:36:46.877929 2040 tablet_replica.cc:330] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: stopping tablet replica
I20260926 21:36:46.878144 2040 raft_consensus.cc:2229] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:36:46.878665 2040 raft_consensus.cc:2258] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:46.879417 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a5dfe8b6a648e590618f51cf537bba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60870
I20260926 21:36:46.879685 2039 tablet_replica.cc:330] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: stopping tablet replica
I20260926 21:36:46.879878 2039 raft_consensus.cc:2229] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:46.880088 2039 raft_consensus.cc:2258] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:46.880594 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a5dfe8b6a648e590618f51cf537bba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60576
I20260926 21:36:46.880875 2041 tablet_replica.cc:330] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: stopping tablet replica
I20260926 21:36:46.881083 2041 raft_consensus.cc:2229] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:46.881315 2041 raft_consensus.cc:2258] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:46.882828 2039 ts_tablet_manager.cc:1905] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:46.885422 2041 ts_tablet_manager.cc:1905] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:46.890417 2040 ts_tablet_manager.cc:1905] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:46.898301 2040 ts_tablet_manager.cc:1918] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:36:46.898504 2040 log.cc:1196] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/63a5dfe8b6a648e590618f51cf537bba
I20260926 21:36:46.898965 2040 ts_tablet_manager.cc:1939] T 63a5dfe8b6a648e590618f51cf537bba P d7cc9fa43b4045a28c208cacab6e8bbd: Deleting consensus metadata
I20260926 21:36:46.900280 1495 catalog_manager.cc:4841] TS d7cc9fa43b4045a28c208cacab6e8bbd (127.0.97.1:33023): tablet 63a5dfe8b6a648e590618f51cf537bba (table test-table [id=4689c9aca7d445a497a008b0ce941420]) successfully deleted
I20260926 21:36:46.903918 2039 ts_tablet_manager.cc:1918] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:36:46.904114 2039 log.cc:1196] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/63a5dfe8b6a648e590618f51cf537bba
I20260926 21:36:46.905018 2039 ts_tablet_manager.cc:1939] T 63a5dfe8b6a648e590618f51cf537bba P fed6fea718054b5481d49a88831153b0: Deleting consensus metadata
I20260926 21:36:46.906141 1498 catalog_manager.cc:4841] TS fed6fea718054b5481d49a88831153b0 (127.0.97.3:44869): tablet 63a5dfe8b6a648e590618f51cf537bba (table test-table [id=4689c9aca7d445a497a008b0ce941420]) successfully deleted
I20260926 21:36:46.909406 2041 ts_tablet_manager.cc:1918] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:36:46.909582 2041 log.cc:1196] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/63a5dfe8b6a648e590618f51cf537bba
I20260926 21:36:46.910017 2041 ts_tablet_manager.cc:1939] T 63a5dfe8b6a648e590618f51cf537bba P 39a77258e7ff44c9bff4db1c48176c6a: Deleting consensus metadata
I20260926 21:36:46.911141 1495 catalog_manager.cc:4841] TS 39a77258e7ff44c9bff4db1c48176c6a (127.0.97.2:44101): tablet 63a5dfe8b6a648e590618f51cf537bba (table test-table [id=4689c9aca7d445a497a008b0ce941420]) successfully deleted
Sep 26 21:36:46 dist-test-slave-lhhq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:36:46.931200 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.901691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.0.97.1:33023)
0926 21:36:46.901892 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.901899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.909055 (+ 7156us) server_negotiation.cc:407] Connection header received
0926 21:36:46.909603 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.909609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.909705 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.909811 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:46.912009 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.913243 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.915587 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.916274 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.916374 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.922529 (+ 6155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:46.922550 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:46.922572 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:46.922620 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:46.927324 (+ 4704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:46.928376 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:46.928389 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:46.928394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:46.928559 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:46.929110 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:46.929116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:46.929119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:46.929597 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:46.929930 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.930562 (+ 632us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.930780 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 21:36:46.932823 1711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58434
I20260926 21:36:46.963295 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.946441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.0.97.62:42675)
0926 21:36:46.946651 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.946657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.946916 (+ 259us) server_negotiation.cc:407] Connection header received
0926 21:36:46.947190 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.947196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.947307 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.947383 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:46.949689 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.950869 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.952752 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.953495 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.953637 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.956637 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:46.956644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:46.956649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:46.956691 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:46.960261 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:46.961072 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:46.961077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:46.961081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:46.961260 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:46.961662 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:46.961667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:46.961670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:46.962052 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:46.962294 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.962704 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.962962 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260926 21:36:46.963898 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34962
I20260926 21:36:46.966821 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1582
I20260926 21:36:46.984638 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:46.967738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.0.97.1:33023)
0926 21:36:46.967939 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:46.967945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:46.968385 (+ 440us) server_negotiation.cc:407] Connection header received
0926 21:36:46.968635 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:46.968641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:46.968718 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:46.968793 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:46.970590 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.971771 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.973802 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:46.974529 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:46.974624 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:46.977739 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:46.977746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:46.977752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:46.977794 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:46.981567 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:46.982414 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:46.982420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:46.982424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:46.982587 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:46.983021 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:46.983030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:46.983034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:46.983372 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:46.983617 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:46.983977 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 21:36:46.984223 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 21:36:46.985711 1711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58446
W20260926 21:36:46.986333 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:36:46.986543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 1582
I20260926 21:36:47.011660 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1722
Sep 26 21:36:47 dist-test-slave-lhhq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:36:47.028641 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:47.012465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.0.97.2:44101)
0926 21:36:47.012634 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:47.012639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:47.012656 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:36:47.012857 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:47.012861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:47.012926 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:47.013009 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:47.014566 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:47.015348 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:47.017361 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:47.017791 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:47.017892 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:47.022857 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:47.022870 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:47.022886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:47.022925 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:47.025998 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:47.026769 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:47.026775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:47.026778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:47.026908 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:47.027273 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:47.027276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:47.027278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:47.027621 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:47.027867 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:47.028161 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 21:36:47.028333 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 21:36:47.029423 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60602
W20260926 21:36:47.030119 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:36:47.030292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 1722
I20260926 21:36:47.051671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1861
Sep 26 21:36:47 dist-test-slave-lhhq krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:36:47.066324 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:47.052484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.0.97.3:44869)
0926 21:36:47.053001 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:47.053005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:47.053021 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:36:47.053074 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:47.053078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:47.053135 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:47.053276 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:47.054418 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:47.055118 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:47.056228 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:47.056639 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:47.056736 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:47.060939 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:47.060954 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:47.060971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:47.061009 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:47.063748 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:47.064402 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:47.064409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:47.064411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:47.064533 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:47.064826 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:47.064829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:47.064831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:47.065175 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:47.065538 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:47.065787 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 21:36:47.065958 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":312,"threads_started":1}
W20260926 21:36:47.066972 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60886
W20260926 21:36:47.067658 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:36:47.067816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 1861
I20260926 21:36:47.085700 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 1479
I20260926 21:36:47.086910 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:47.199147 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 1479
2026-09-26T21:36:47Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4651 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:36:47 dist-test-slave-lhhq krb5kdc[2084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:36:47 dist-test-slave-lhhq krb5kdc[2084](info): set up 2 sockets
Sep 26 21:36:47 dist-test-slave-lhhq krb5kdc[2084](info): commencing operation
krb5kdc: starting...
W20260926 21:36:48.301373 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:36:48 dist-test-slave-lhhq krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:36:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:36:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:36:48.492902 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34987
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:48.808045 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:48.808350 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:48.808506 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:48.815815 2097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:36:48.815932 2097 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 21:36:48.816010 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:48.816135 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:48.816211 2097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:36:48.816320 2097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:36:48.826546 2097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34987
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:48.827837 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:48.829213 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:48.840708 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:48.840842 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:48.840874 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:48.841831 2097 server_base.cc:1034] running on GCE node
I20260926 21:36:48.842635 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:48.843987 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:48.845263 2097 hybrid_clock.cc:648] HybridClock initialized: now 1790458608845163 us; error 35 us; skew 500 ppm
I20260926 21:36:48.845574 2097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:48 dist-test-slave-lhhq krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:48.848482 2097 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:36:48.850013 2097 webserver.cc:460] Webserver started at http://127.0.97.62:39757/ using document root <none> and password file <none>
I20260926 21:36:48.850711 2097 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:48.850839 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:48.851145 2097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:48.854060 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "17056a7b8ff642449b31b9374be663a7"
format_stamp: "Formatted at 2026-09-26 21:36:48 on dist-test-slave-lhhq"
server_key: "5e7d7494cd8b571f4c1d6b8a2e19bb0b"
server_key_iv: "0ba5f3dfd377b13cca3d1289797769a7"
server_key_version: "encryptionkey@0"
I20260926 21:36:48.854800 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "17056a7b8ff642449b31b9374be663a7"
format_stamp: "Formatted at 2026-09-26 21:36:48 on dist-test-slave-lhhq"
server_key: "5e7d7494cd8b571f4c1d6b8a2e19bb0b"
server_key_iv: "0ba5f3dfd377b13cca3d1289797769a7"
server_key_version: "encryptionkey@0"
I20260926 21:36:48.859813 2097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:36:48.863358 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:48.864723 2097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:36:48.864881 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "17056a7b8ff642449b31b9374be663a7"
format_stamp: "Formatted at 2026-09-26 21:36:48 on dist-test-slave-lhhq"
server_key: "5e7d7494cd8b571f4c1d6b8a2e19bb0b"
server_key_iv: "0ba5f3dfd377b13cca3d1289797769a7"
server_key_version: "encryptionkey@0"
I20260926 21:36:48.865095 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:48.879949 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:48.884152 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:48.884424 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:48.905242 2097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34987
I20260926 21:36:48.905303 2173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34987 every 8 connection(s)
I20260926 21:36:48.906623 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:36:48.910295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 2097
I20260926 21:36:48.910487 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:48.910512 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:36:48.911137 388 external_mini_cluster.cc:1428] Setting key 74575ebee7a17d35663741a004339121
I20260926 21:36:48.924906 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7: Bootstrap starting.
Sep 26 21:36:48 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:48.928736 2174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:48.929947 2174 log.cc:824] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:48.932952 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7: No bootstrap required, opened a new log
I20260926 21:36:48.934084 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:48.913101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.0.97.62:34987)
0926 21:36:48.913988 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:48.914002 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:48.914049 (+ 47us) server_negotiation.cc:407] Connection header received
0926 21:36:48.915724 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:48.915753 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:48.916255 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:48.916734 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:48.918113 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:48.920094 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:48.921406 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:48.922545 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:48.922749 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:48.926659 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:48.926668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:48.926687 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:48.926737 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:48.929843 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:48.930616 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:48.930623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:48.930632 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:48.930803 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:48.931198 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:48.931202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:48.931204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:48.931735 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:48.932030 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:48.932491 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 21:36:48.932921 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":336,"threads_started":1}
I20260926 21:36:48.941093 2174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17056a7b8ff642449b31b9374be663a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } }
I20260926 21:36:48.941610 2174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:48.941721 2174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17056a7b8ff642449b31b9374be663a7, State: Initialized, Role: FOLLOWER
I20260926 21:36:48.942278 2174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17056a7b8ff642449b31b9374be663a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } }
I20260926 21:36:48.942487 2174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:36:48.942571 2174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:36:48.942693 2174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:48.945106 2174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17056a7b8ff642449b31b9374be663a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } }
I20260926 21:36:48.945644 2174 leader_election.cc:302] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17056a7b8ff642449b31b9374be663a7; no voters:
I20260926 21:36:48.946132 2174 leader_election.cc:288] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:36:48.946281 2179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:48.946990 2179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [term 1 LEADER]: Becoming Leader. State: Replica: 17056a7b8ff642449b31b9374be663a7, State: Running, Role: LEADER
I20260926 21:36:48.947496 2179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17056a7b8ff642449b31b9374be663a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } }
I20260926 21:36:48.948253 2174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:36:48.950359 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17056a7b8ff642449b31b9374be663a7. Latest consensus state: current_term: 1 leader_uuid: "17056a7b8ff642449b31b9374be663a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17056a7b8ff642449b31b9374be663a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } } }
I20260926 21:36:48.950470 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17056a7b8ff642449b31b9374be663a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17056a7b8ff642449b31b9374be663a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34987 } } }
I20260926 21:36:48.950810 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:48.950809 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:48.952790 2187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:36:48.958294 2187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:36:48.968585 2187 catalog_manager.cc:1347] Generated new cluster ID: f1c35c7fc73646d2a1451ba8fdb86b8d
I20260926 21:36:48.968721 2187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:36:49.008456 2187 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:36:49.010429 2187 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:36:49.023871 2187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7: Generated new TSK 0
I20260926 21:36:49.024768 2187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:36:49.076470 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34987
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:49.412196 2200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:49.412492 2200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:49.412595 2200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:49.419854 2200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:49.419978 2200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:49.420065 2200 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 21:36:49.420118 2200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:36:49.420181 2200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:36:49.430259 2200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:49.431650 2200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:49.432955 2200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:49.446442 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:49.446400 2208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:49.446491 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:49.447501 2200 server_base.cc:1034] running on GCE node
I20260926 21:36:49.448225 2200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:49.449502 2200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:49.450748 2200 hybrid_clock.cc:648] HybridClock initialized: now 1790458609450695 us; error 59 us; skew 500 ppm
I20260926 21:36:49.451135 2200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:49 dist-test-slave-lhhq krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:49.454489 2200 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:36:49.456019 2200 webserver.cc:460] Webserver started at http://127.0.97.1:32911/ using document root <none> and password file <none>
I20260926 21:36:49.456766 2200 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:49.456921 2200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:49.457312 2200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:49.460368 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "9ff87a116bbd424c896d96c50e4e008d"
format_stamp: "Formatted at 2026-09-26 21:36:49 on dist-test-slave-lhhq"
server_key: "218daef0b6f4f5488d0f791b90125003"
server_key_iv: "980bdacfd37165d4f63c789a71087339"
server_key_version: "encryptionkey@0"
I20260926 21:36:49.461268 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9ff87a116bbd424c896d96c50e4e008d"
format_stamp: "Formatted at 2026-09-26 21:36:49 on dist-test-slave-lhhq"
server_key: "218daef0b6f4f5488d0f791b90125003"
server_key_iv: "980bdacfd37165d4f63c789a71087339"
server_key_version: "encryptionkey@0"
I20260926 21:36:49.466506 2200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:36:49.470326 2215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:49.471761 2200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:36:49.471962 2200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "9ff87a116bbd424c896d96c50e4e008d"
format_stamp: "Formatted at 2026-09-26 21:36:49 on dist-test-slave-lhhq"
server_key: "218daef0b6f4f5488d0f791b90125003"
server_key_iv: "980bdacfd37165d4f63c789a71087339"
server_key_version: "encryptionkey@0"
I20260926 21:36:49.472170 2200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:49.484786 2200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:49.488902 2200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:49.489256 2200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:49.491973 2200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:49.492118 2200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:49.492293 2200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:49.492398 2200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:36:49 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:49.516170 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:49.492368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51623 (local address 127.0.97.62:34987)
0926 21:36:49.492908 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:49.492914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:49.494219 (+ 1305us) server_negotiation.cc:407] Connection header received
0926 21:36:49.497263 (+ 3044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:49.497269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:49.497349 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:49.497507 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:49.499862 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.500917 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:49.503334 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.503958 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:49.504093 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:49.509299 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:49.509305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:49.509308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:49.509343 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:49.512451 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:49.513452 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:49.513462 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:49.513465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:49.513577 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:49.514275 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:49.514283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:49.514288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:49.514659 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:49.514816 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:49.515540 (+ 724us) server_negotiation.cc:299] Negotiation successful
0926 21:36:49.515754 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":242,"threads_started":1}
I20260926 21:36:49.517584 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:49.493061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.97.1:51623)
0926 21:36:49.494019 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:49.494070 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:49.496871 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:49.497814 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:49.497833 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:49.498693 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:49.499610 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:49.499641 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.501071 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:49.501077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:49.503183 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:49.503195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.504210 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:49.504217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:49.504447 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:49.505367 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:49.505399 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:49.508926 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:49.512603 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:49.512613 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:49.512635 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:49.513305 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:49.513757 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:49.513762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:49.513766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:49.514111 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:49.514803 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:49.514811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:49.515181 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 21:36:49.515456 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 21:36:49.515997 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":338,"threads_started":1}
I20260926 21:36:49.540759 2200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36187
I20260926 21:36:49.540776 2335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36187 every 8 connection(s)
I20260926 21:36:49.542534 2200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:36:49.543987 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:49.527642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.97.1:33941)
0926 21:36:49.528389 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:49.528415 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:49.528606 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:49.528948 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:49.528955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:49.529796 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:49.530281 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:49.530297 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.531864 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:49.531870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:49.534074 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:49.534100 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.535067 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:49.535077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:49.535328 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:49.536132 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:49.536161 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:49.537375 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:49.541289 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:49.541293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:49.541296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:49.541675 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:49.542075 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:49.542078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:49.542080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:49.542191 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:49.542796 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:49.542799 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:49.542956 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 21:36:49.543111 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 21:36:49.543503 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":372,"threads_started":1}
I20260926 21:36:49.544068 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:49.527638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33941 (local address 127.0.97.62:34987)
0926 21:36:49.527853 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:49.527862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:49.528528 (+ 666us) server_negotiation.cc:407] Connection header received
0926 21:36:49.528734 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:49.528738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:49.528798 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:49.528889 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:49.530523 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.531660 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:49.534260 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.534824 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:49.534970 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:49.537607 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:49.537614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:49.537619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:49.537666 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:49.541129 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:49.541805 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:49.541811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:49.541814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:49.541965 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:49.542302 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:49.542307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:49.542318 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:49.542682 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:49.542829 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:49.543262 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 21:36:49.543605 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:36:49.544782 2336 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34987
I20260926 21:36:49.545013 2336 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:49.545923 2336 heartbeater.cc:511] Master 127.0.97.62:34987 requested a full tablet report, sending...
I20260926 21:36:49.547127 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 2200
I20260926 21:36:49.547391 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:36:49.548228 2129 ts_manager.cc:194] Registered new tserver with Master: 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187)
I20260926 21:36:49.548297 388 external_mini_cluster.cc:1428] Setting key 0ba784da9cdedf62a7255331ba387a29
I20260926 21:36:49.549912 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51623
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:36:49.588197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34987
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:49.909853 2340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:49.910110 2340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:49.910296 2340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:49.917606 2340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:49.917704 2340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:49.917893 2340 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 21:36:49.917990 2340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:36:49.918111 2340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:36:49.928292 2340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:49.929739 2340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:49.931044 2340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:49.943831 2345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:49.943866 2346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:49.943835 2348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:49.946743 2340 server_base.cc:1034] running on GCE node
I20260926 21:36:49.947350 2340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:49.948280 2340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:49.949494 2340 hybrid_clock.cc:648] HybridClock initialized: now 1790458609949450 us; error 40 us; skew 500 ppm
I20260926 21:36:49.949805 2340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:49 dist-test-slave-lhhq krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:49.952936 2340 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:36:49.954454 2340 webserver.cc:460] Webserver started at http://127.0.97.2:36231/ using document root <none> and password file <none>
I20260926 21:36:49.955094 2340 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:49.955188 2340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:49.955431 2340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:49.958312 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "0e1bb9b74bbd4acca5da79884c7d9273"
format_stamp: "Formatted at 2026-09-26 21:36:49 on dist-test-slave-lhhq"
server_key: "ee145409e531a381cc07e6a843be3e2c"
server_key_iv: "e35058e9ab44e1f23da249d165e82140"
server_key_version: "encryptionkey@0"
I20260926 21:36:49.959035 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0e1bb9b74bbd4acca5da79884c7d9273"
format_stamp: "Formatted at 2026-09-26 21:36:49 on dist-test-slave-lhhq"
server_key: "ee145409e531a381cc07e6a843be3e2c"
server_key_iv: "e35058e9ab44e1f23da249d165e82140"
server_key_version: "encryptionkey@0"
I20260926 21:36:49.964143 2340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:36:49.967731 2355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:49.969141 2340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:36:49.969385 2340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "0e1bb9b74bbd4acca5da79884c7d9273"
format_stamp: "Formatted at 2026-09-26 21:36:49 on dist-test-slave-lhhq"
server_key: "ee145409e531a381cc07e6a843be3e2c"
server_key_iv: "e35058e9ab44e1f23da249d165e82140"
server_key_version: "encryptionkey@0"
I20260926 21:36:49.969650 2340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:49.983006 2340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:49.987325 2340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:49.987624 2340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:49.990418 2340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:49.990563 2340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:49.990707 2340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:49.990792 2340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:36:50 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:50.018708 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:49.991493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.97.2:43439)
0926 21:36:49.992344 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:49.992396 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:49.994853 (+ 2457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:49.995701 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:49.995719 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:49.996722 (+ 1003us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:49.997930 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:49.997967 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.999083 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:49.999091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.002035 (+ 2944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.002054 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.003126 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.003137 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.003538 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.004365 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:50.004415 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:50.008594 (+ 4179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:50.013126 (+ 4532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.013144 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.013168 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.014049 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.014549 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.014554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.014558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.014917 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.015646 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:50.015655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:50.016054 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.016371 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.016993 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":394,"threads_started":1}
I20260926 21:36:50.019783 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:49.990885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43439 (local address 127.0.97.62:34987)
0926 21:36:49.991101 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:49.991111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:49.992527 (+ 1416us) server_negotiation.cc:407] Connection header received
0926 21:36:49.995170 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:49.995176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:49.995271 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:49.995395 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:49.998173 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:49.998907 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.002233 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.002792 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.002952 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.008961 (+ 6009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:50.008973 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:50.008981 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:50.009030 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:50.012907 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.014235 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.014243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.014248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.014428 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.015112 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.015117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.015120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.015517 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:50.015664 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.019326 (+ 3662us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.019538 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:36:50.039454 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.025757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.97.2:55193)
0926 21:36:50.026384 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:50.026400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:50.026571 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:50.026907 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:50.026912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:50.027355 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:50.027621 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.027630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.028729 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.028734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.030226 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.030243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.031001 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.031007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.031179 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.031905 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:50.031922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:50.032650 (+ 728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:50.036346 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.036352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.036357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.037064 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.037654 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.037658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.037661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.037777 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.038610 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:50.038614 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:50.038767 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.038890 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.039148 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":368,"threads_started":1}
I20260926 21:36:50.039824 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.025684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55193 (local address 127.0.97.62:34987)
0926 21:36:50.025846 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.025850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.026497 (+ 647us) server_negotiation.cc:407] Connection header received
0926 21:36:50.026702 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.026714 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.026775 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.026878 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:50.027766 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.028589 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.030394 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.030811 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.030955 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.032843 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:50.032848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:50.032851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:50.032882 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:50.036122 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.037310 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.037320 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.037326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.037516 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.037893 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.037900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.037906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.038465 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:50.038645 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.039054 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.039350 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:36:50.047987 2340 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36219
I20260926 21:36:50.048022 2474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36219 every 8 connection(s)
I20260926 21:36:50.049428 2340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:36:50.051491 2475 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34987
I20260926 21:36:50.051718 2475 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:50.052376 2475 heartbeater.cc:511] Master 127.0.97.62:34987 requested a full tablet report, sending...
I20260926 21:36:50.053776 2129 ts_manager.cc:194] Registered new tserver with Master: 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219)
I20260926 21:36:50.054557 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43439
I20260926 21:36:50.056020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 2340
I20260926 21:36:50.056198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:36:50.056615 388 external_mini_cluster.cc:1428] Setting key c43e7e23cf1b89abe62dcc8269941406
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:36:50.096856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34987
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:50.417510 2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:50.417795 2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:50.417907 2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:50.425395 2479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:50.425509 2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:50.425595 2479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 21:36:50.425647 2479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:36:50.425716 2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:36:50.435505 2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:50.436756 2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:50.438123 2479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:50.451128 2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:50.451151 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:50.451370 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:50.451984 2479 server_base.cc:1034] running on GCE node
I20260926 21:36:50.452651 2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:50.453723 2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:50.454910 2479 hybrid_clock.cc:648] HybridClock initialized: now 1790458610454851 us; error 68 us; skew 500 ppm
I20260926 21:36:50.455222 2479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:50 dist-test-slave-lhhq krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:50.466661 2479 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:36:50.468390 2479 webserver.cc:460] Webserver started at http://127.0.97.3:33539/ using document root <none> and password file <none>
I20260926 21:36:50.469167 2479 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:50.469377 2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:50.469687 2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:50.472615 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "b6d6d31ed2fa4e8793b81619badaf61c"
format_stamp: "Formatted at 2026-09-26 21:36:50 on dist-test-slave-lhhq"
server_key: "3554b7e1875b26970c402bab5108c29d"
server_key_iv: "355fa811cc594b743ee2a69340c92ef5"
server_key_version: "encryptionkey@0"
I20260926 21:36:50.473482 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b6d6d31ed2fa4e8793b81619badaf61c"
format_stamp: "Formatted at 2026-09-26 21:36:50 on dist-test-slave-lhhq"
server_key: "3554b7e1875b26970c402bab5108c29d"
server_key_iv: "355fa811cc594b743ee2a69340c92ef5"
server_key_version: "encryptionkey@0"
I20260926 21:36:50.478758 2479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:36:50.482034 2494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:50.483294 2479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:36:50.483471 2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "b6d6d31ed2fa4e8793b81619badaf61c"
format_stamp: "Formatted at 2026-09-26 21:36:50 on dist-test-slave-lhhq"
server_key: "3554b7e1875b26970c402bab5108c29d"
server_key_iv: "355fa811cc594b743ee2a69340c92ef5"
server_key_version: "encryptionkey@0"
I20260926 21:36:50.483671 2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:50.500799 2479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:50.505300 2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:50.505607 2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:50.508247 2479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:50.508396 2479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:50.508529 2479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:50.508616 2479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:36:50 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:50.528784 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.508734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55585 (local address 127.0.97.62:34987)
0926 21:36:50.508995 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.509005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.510266 (+ 1261us) server_negotiation.cc:407] Connection header received
0926 21:36:50.512227 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.512233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.512326 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.512442 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:50.514176 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.514902 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.516503 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.516907 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.517039 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.521638 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:50.521643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:50.521646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:50.521679 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:50.524942 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.526282 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.526291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.526296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.526470 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.527071 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.527076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.527080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.527355 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:50.527507 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.528213 (+ 706us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.528428 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 21:36:50.530221 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.509154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.97.3:55585)
0926 21:36:50.510069 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:50.510125 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:50.511962 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:50.512656 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:50.512663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:50.513385 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:50.514011 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.514028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.515062 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.515073 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.516347 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.516358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.517133 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.517140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.517399 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.518304 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:50.518339 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:50.521334 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:50.525149 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.525161 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.525222 (+ 61us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.526059 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.526583 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.526589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.526593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.526908 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.527482 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:50.527491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:50.527845 (+ 354us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.528105 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.528664 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":398,"threads_started":1}
I20260926 21:36:50.552728 2336 heartbeater.cc:503] Master 127.0.97.62:34987 was elected leader, sending a full tablet report...
I20260926 21:36:50.556052 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.540760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49033 (local address 127.0.97.62:34987)
0926 21:36:50.540940 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.540946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.541164 (+ 218us) server_negotiation.cc:407] Connection header received
0926 21:36:50.541485 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.541493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.541573 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.541649 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:50.543087 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.544171 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.546409 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.547074 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.547189 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.549756 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:50.549761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:50.549764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:50.549791 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:50.553267 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.554039 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.554045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.554048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.554183 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.554640 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.554645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.554648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.554929 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:50.555082 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.555490 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.555695 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 21:36:50.556140 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.540220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.97.3:49033)
0926 21:36:50.541058 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:50.541082 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:50.541313 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:50.541862 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:50.541867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:50.542480 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:50.542905 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.542920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.544378 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.544383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.546246 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.546259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.547424 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.547430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.547582 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.548308 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:50.548338 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:50.549548 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:50.553453 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.553457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.553460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.553923 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.554288 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.554294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.554297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.554426 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.555051 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:50.555055 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:50.555239 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.555369 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.555695 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":597,"threads_started":1}
I20260926 21:36:50.570303 2479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44527
I20260926 21:36:50.570364 2613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44527 every 8 connection(s)
I20260926 21:36:50.571542 2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:36:50.573535 2614 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34987
I20260926 21:36:50.573747 2614 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:50.574381 2614 heartbeater.cc:511] Master 127.0.97.62:34987 requested a full tablet report, sending...
I20260926 21:36:50.575158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 2479
I20260926 21:36:50.575340 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:36:50.575733 388 external_mini_cluster.cc:1428] Setting key 1f7e9dcbad710cbd266a01817b22e8b7
I20260926 21:36:50.575714 2129 ts_manager.cc:194] Registered new tserver with Master: b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527)
I20260926 21:36:50.576678 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55585
I20260926 21:36:50.578627 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:36:50 dist-test-slave-lhhq krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 21:36:50 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:50.624750 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.609946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.0.97.62:34987)
0926 21:36:50.610176 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.610185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.610397 (+ 212us) server_negotiation.cc:407] Connection header received
0926 21:36:50.610633 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.610638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.610719 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.610837 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:50.612337 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.613086 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.614458 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.614902 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.615049 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.619119 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:50.619125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:50.619128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:50.619164 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:50.621901 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.622596 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.622604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.622608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.622755 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.623172 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.623179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.623183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.623455 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:50.623719 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.624062 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.624308 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:36:50.628818 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:36:50.635720 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:36:50.660162 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.646963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44527 (local address 127.0.0.1:46200)
0926 21:36:50.647785 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:50.647831 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:50.648049 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:50.653076 (+ 5027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:50.653081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:50.653115 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:50.653562 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.653574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.655694 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.655699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.657464 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.657480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.659285 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.659292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.659486 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.659539 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.659655 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.659848 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":18,"thread_start_us":416,"threads_started":1}
I20260926 21:36:50.661038 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.647844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.0.97.2:36219)
0926 21:36:50.650400 (+ 2556us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.650413 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.650449 (+ 36us) server_negotiation.cc:407] Connection header received
0926 21:36:50.650566 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.650577 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.651525 (+ 948us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.651757 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:50.653304 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.654897 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.658315 (+ 3418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.659725 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.659839 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.660032 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.660574 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.660766 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2346,"thread_start_us":2201,"threads_started":1}
I20260926 21:36:50.661059 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.647102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.0.97.3:44527)
0926 21:36:50.652497 (+ 5395us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.652505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.652535 (+ 30us) server_negotiation.cc:407] Connection header received
0926 21:36:50.652623 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.652633 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.652864 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.653742 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:50.654026 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.655543 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.657671 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.659062 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.660412 (+ 1350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.660626 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.660700 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.660839 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5248,"thread_start_us":5050,"threads_started":1}
I20260926 21:36:50.661834 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.647304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36219 (local address 127.0.0.1:33394)
0926 21:36:50.648911 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:50.648933 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:50.649066 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:50.652492 (+ 3426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:50.652499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:50.652532 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:50.653056 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.653070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.655073 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.655078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.658167 (+ 3089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.658178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.660176 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.660183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.660358 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.660407 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.661329 (+ 922us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.661474 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1476,"mutex_wait_us":26,"thread_start_us":1390,"threads_started":1}
I20260926 21:36:50.664732 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 93a0d15e6e7d4aadbb9012c82aaef6e9 (DEFAULT_TABLE table=test-table [id=a9bbb3a6421c4ff89b36caaf530b06bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:50.665486 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.650592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.0.97.1:36187)
0926 21:36:50.657860 (+ 7268us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.657869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.657894 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:36:50.657980 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.657987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.658204 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.658392 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:50.659039 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.660440 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.662813 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.664180 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.664311 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.664499 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.665034 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.665222 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7108,"thread_start_us":6978,"threads_started":1}
I20260926 21:36:50.665421 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.650102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36187 (local address 127.0.0.1:37358)
0926 21:36:50.656672 (+ 6570us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:50.656685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:50.656770 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:50.658390 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:50.658394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:50.658418 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:50.658796 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.658807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.660587 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.660592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.662644 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.662659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.664636 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.664642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.664835 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.664870 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.664966 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.665093 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6468,"thread_start_us":6316,"threads_started":1}
I20260926 21:36:50.665030 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 93a0d15e6e7d4aadbb9012c82aaef6e9 (DEFAULT_TABLE table=test-table [id=a9bbb3a6421c4ff89b36caaf530b06bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:50.666329 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93a0d15e6e7d4aadbb9012c82aaef6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:50.666374 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93a0d15e6e7d4aadbb9012c82aaef6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:50.669389 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 93a0d15e6e7d4aadbb9012c82aaef6e9 (DEFAULT_TABLE table=test-table [id=a9bbb3a6421c4ff89b36caaf530b06bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:50.671363 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93a0d15e6e7d4aadbb9012c82aaef6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:50.680301 2634 tablet_bootstrap.cc:492] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: Bootstrap starting.
I20260926 21:36:50.682044 2633 tablet_bootstrap.cc:492] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: Bootstrap starting.
I20260926 21:36:50.683835 2634 tablet_bootstrap.cc:654] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:50.684798 2634 log.cc:824] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:50.685349 2633 tablet_bootstrap.cc:654] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:50.685956 2635 tablet_bootstrap.cc:492] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: Bootstrap starting.
I20260926 21:36:50.686527 2633 log.cc:824] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:50.690618 2635 tablet_bootstrap.cc:654] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:50.691576 2634 tablet_bootstrap.cc:492] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: No bootstrap required, opened a new log
I20260926 21:36:50.691846 2633 tablet_bootstrap.cc:492] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: No bootstrap required, opened a new log
I20260926 21:36:50.691823 2635 log.cc:824] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: Log is configured to *not* fsync() on all Append() calls
I20260926 21:36:50.691859 2634 ts_tablet_manager.cc:1397] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 21:36:50.692094 2633 ts_tablet_manager.cc:1397] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 21:36:50.694725 2635 tablet_bootstrap.cc:492] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: No bootstrap required, opened a new log
I20260926 21:36:50.694971 2635 ts_tablet_manager.cc:1397] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 21:36:50.699577 2634 raft_consensus.cc:348] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } }
I20260926 21:36:50.699994 2634 raft_consensus.cc:374] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:50.700093 2634 raft_consensus.cc:729] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1bb9b74bbd4acca5da79884c7d9273, State: Initialized, Role: FOLLOWER
I20260926 21:36:50.700693 2634 consensus_queue.cc:260] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } }
I20260926 21:36:50.701793 2475 heartbeater.cc:503] Master 127.0.97.62:34987 was elected leader, sending a full tablet report...
I20260926 21:36:50.702309 2634 ts_tablet_manager.cc:1428] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260926 21:36:50.702339 2633 raft_consensus.cc:348] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } }
I20260926 21:36:50.702749 2633 raft_consensus.cc:374] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:50.702859 2633 raft_consensus.cc:729] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d6d31ed2fa4e8793b81619badaf61c, State: Initialized, Role: FOLLOWER
I20260926 21:36:50.703390 2633 consensus_queue.cc:260] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } }
I20260926 21:36:50.704435 2614 heartbeater.cc:503] Master 127.0.97.62:34987 was elected leader, sending a full tablet report...
I20260926 21:36:50.705044 2633 ts_tablet_manager.cc:1428] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260926 21:36:50.706141 2635 raft_consensus.cc:348] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } }
I20260926 21:36:50.706524 2635 raft_consensus.cc:374] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:50.706643 2635 raft_consensus.cc:729] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff87a116bbd424c896d96c50e4e008d, State: Initialized, Role: FOLLOWER
I20260926 21:36:50.707233 2635 consensus_queue.cc:260] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } }
I20260926 21:36:50.708921 2635 ts_tablet_manager.cc:1428] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260926 21:36:50.770095 2639 raft_consensus.cc:482] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:36:50.770414 2639 raft_consensus.cc:504] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } }
I20260926 21:36:50.772100 2639 leader_election.cc:288] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527), 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187)
I20260926 21:36:50.779081 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.772480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58531 (local address 127.0.97.3:44527)
0926 21:36:50.772693 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.772701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.773543 (+ 842us) server_negotiation.cc:407] Connection header received
0926 21:36:50.773699 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.773704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.773783 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.773883 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:50.774567 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.775320 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.777400 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.778136 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.778279 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.778391 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.778689 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.778802 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 21:36:50.779220 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.772474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56429 (local address 127.0.97.1:36187)
0926 21:36:50.772671 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.772680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.773682 (+ 1002us) server_negotiation.cc:407] Connection header received
0926 21:36:50.773846 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.773849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.773924 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.774021 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:50.774623 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.775314 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.777409 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.778163 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.778305 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.778415 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.778844 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.778978 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:36:50.779270 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.772487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44527 (local address 127.0.97.2:58531)
0926 21:36:50.773419 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:50.773441 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:50.773567 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:50.773894 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:50.773898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:50.773919 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:50.774427 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.774440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.775505 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.775511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.777256 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.777268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.778387 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.778393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.778539 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.778561 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.778667 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.778791 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":3,"spinlock_wait_cycles":32000,"thread_start_us":468,"threads_started":1}
I20260926 21:36:50.779350 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.772621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36187 (local address 127.0.97.2:56429)
0926 21:36:50.773574 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:50.773591 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:50.773729 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:50.774038 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:50.774042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:50.774100 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:50.774461 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.774469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.775492 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.775502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.777270 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.777281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.778443 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.778450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.778650 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.778673 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.778806 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.778928 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":43,"thread_start_us":649,"threads_started":1}
I20260926 21:36:50.780191 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a0d15e6e7d4aadbb9012c82aaef6e9" candidate_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" is_pre_election: true
I20260926 21:36:50.780673 2569 raft_consensus.cc:2454] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1bb9b74bbd4acca5da79884c7d9273 in term 0.
I20260926 21:36:50.780568 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a0d15e6e7d4aadbb9012c82aaef6e9" candidate_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff87a116bbd424c896d96c50e4e008d" is_pre_election: true
I20260926 21:36:50.781032 2286 raft_consensus.cc:2454] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1bb9b74bbd4acca5da79884c7d9273 in term 0.
I20260926 21:36:50.781351 2357 leader_election.cc:302] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1bb9b74bbd4acca5da79884c7d9273, b6d6d31ed2fa4e8793b81619badaf61c; no voters:
I20260926 21:36:50.781729 2639 raft_consensus.cc:2781] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:36:50.781875 2639 raft_consensus.cc:482] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:36:50.781977 2639 raft_consensus.cc:3037] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:50.784040 2639 raft_consensus.cc:504] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } }
I20260926 21:36:50.784614 2639 leader_election.cc:288] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [CANDIDATE]: Term 1 election: Requested vote from peers b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527), 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187)
I20260926 21:36:50.785024 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a0d15e6e7d4aadbb9012c82aaef6e9" candidate_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d6d31ed2fa4e8793b81619badaf61c"
I20260926 21:36:50.785164 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a0d15e6e7d4aadbb9012c82aaef6e9" candidate_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff87a116bbd424c896d96c50e4e008d"
I20260926 21:36:50.785252 2569 raft_consensus.cc:3037] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:50.785405 2286 raft_consensus.cc:3037] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:50.787575 2569 raft_consensus.cc:2454] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1bb9b74bbd4acca5da79884c7d9273 in term 1.
I20260926 21:36:50.787626 2286 raft_consensus.cc:2454] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1bb9b74bbd4acca5da79884c7d9273 in term 1.
I20260926 21:36:50.787995 2357 leader_election.cc:302] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1bb9b74bbd4acca5da79884c7d9273, b6d6d31ed2fa4e8793b81619badaf61c; no voters:
I20260926 21:36:50.788293 2639 raft_consensus.cc:2781] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:50.788754 2639 raft_consensus.cc:686] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 LEADER]: Becoming Leader. State: Replica: 0e1bb9b74bbd4acca5da79884c7d9273, State: Running, Role: LEADER
I20260926 21:36:50.789463 2639 consensus_queue.cc:237] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } }
I20260926 21:36:50.794446 2128 catalog_manager.cc:5495] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } health_report { overall_health: UNKNOWN } } }
W20260926 21:36:50.800940 2337 tablet.cc:2369] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:36:50.801506 2476 tablet.cc:2369] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:36:50.823089 2615 tablet.cc:2369] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 21:36:50 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:50.836838 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.823897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.0.97.62:34987)
0926 21:36:50.824084 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:50.824090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:50.824619 (+ 529us) server_negotiation.cc:407] Connection header received
0926 21:36:50.824819 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:50.824824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:50.824885 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:50.824986 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:50.826197 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.826910 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.828157 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.828573 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.828721 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.832004 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:50.832010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:50.832021 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:50.832047 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:50.834297 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.834926 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.834933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.834940 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.835064 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:50.835407 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:50.835415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:50.835418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:50.835664 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:50.835817 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:50.836197 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 21:36:50.836510 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":66}
I20260926 21:36:50.836869 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:50.823913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34987 (local address 127.0.0.1:59556)
0926 21:36:50.824511 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:50.824527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:50.824697 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:50.825007 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:50.825011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:50.825791 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:50.826057 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.826066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.827036 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.827043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.828015 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:50.828027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:50.828770 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:50.828776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:50.828945 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:50.829619 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:50.829640 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:50.831841 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:50.834418 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.834431 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.834438 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.834816 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.835165 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:50.835172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:50.835175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:50.835309 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:50.835792 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:50.835799 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:50.835962 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 21:36:50.836115 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 21:36:50.836395 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":338,"threads_started":1}
I20260926 21:36:50.841935 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59556:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 21:36:50.842958 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:36:50.849498 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9b0bf552324d76a057d598dcb383a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d874aa08d5f74814b3f2ae5044ee5ef4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:36:50.849678 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9b0bf552324d76a057d598dcb383a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d874aa08d5f74814b3f2ae5044ee5ef4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:36:50.850454 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e9b0bf552324d76a057d598dcb383a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:50.850478 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e9b0bf552324d76a057d598dcb383a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:50.853327 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9b0bf552324d76a057d598dcb383a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d874aa08d5f74814b3f2ae5044ee5ef4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:36:50.854195 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e9b0bf552324d76a057d598dcb383a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:50.856160 2633 tablet_bootstrap.cc:492] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: Bootstrap starting.
I20260926 21:36:50.857502 2633 tablet_bootstrap.cc:654] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:50.860098 2635 tablet_bootstrap.cc:492] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: Bootstrap starting.
I20260926 21:36:50.861631 2635 tablet_bootstrap.cc:654] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:50.861711 2633 tablet_bootstrap.cc:492] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: No bootstrap required, opened a new log
I20260926 21:36:50.861842 2634 tablet_bootstrap.cc:492] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: Bootstrap starting.
I20260926 21:36:50.861864 2633 ts_tablet_manager.cc:1397] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 21:36:50.862527 2633 raft_consensus.cc:348] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:50.862813 2633 raft_consensus.cc:374] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:50.862921 2633 raft_consensus.cc:729] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d6d31ed2fa4e8793b81619badaf61c, State: Initialized, Role: FOLLOWER
I20260926 21:36:50.863201 2633 consensus_queue.cc:260] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:50.863504 2634 tablet_bootstrap.cc:654] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:50.863652 2640 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:50.863997 2640 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:50.864272 2635 tablet_bootstrap.cc:492] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: No bootstrap required, opened a new log
I20260926 21:36:50.864459 2635 ts_tablet_manager.cc:1397] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 21:36:50.864645 2633 ts_tablet_manager.cc:1428] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 21:36:50.865307 2635 raft_consensus.cc:348] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:50.865481 2635 raft_consensus.cc:374] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:50.865569 2635 raft_consensus.cc:729] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff87a116bbd424c896d96c50e4e008d, State: Initialized, Role: FOLLOWER
I20260926 21:36:50.865784 2635 consensus_queue.cc:260] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:50.866256 2634 tablet_bootstrap.cc:492] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: No bootstrap required, opened a new log
I20260926 21:36:50.866397 2634 ts_tablet_manager.cc:1397] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260926 21:36:50.866225 2641 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:50.866624 2641 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:50.867319 2635 ts_tablet_manager.cc:1428] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 21:36:50.867174 2634 raft_consensus.cc:348] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:50.867405 2634 raft_consensus.cc:374] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:50.867499 2634 raft_consensus.cc:729] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1bb9b74bbd4acca5da79884c7d9273, State: Initialized, Role: FOLLOWER
I20260926 21:36:50.867751 2634 consensus_queue.cc:260] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:50.868166 2639 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:50.868446 2639 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:50.868876 2634 ts_tablet_manager.cc:1428] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 21:36:51.012380 2640 raft_consensus.cc:482] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:36:51.012574 2640 raft_consensus.cc:504] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.014127 2640 leader_election.cc:288] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187), 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219)
I20260926 21:36:51.020538 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.014502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36187 (local address 127.0.97.3:60893)
0926 21:36:51.014678 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:51.014701 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:51.014835 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:51.015353 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:51.015358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:51.015386 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:51.015597 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.015610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.016868 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.016873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.018402 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.018413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.019680 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.019686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.019881 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.019930 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 21:36:51.020028 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 21:36:51.020190 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":22}
I20260926 21:36:51.020630 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.014499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60893 (local address 127.0.97.1:36187)
0926 21:36:51.014662 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:51.014667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:51.014772 (+ 105us) server_negotiation.cc:407] Connection header received
0926 21:36:51.015032 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:51.015037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:51.015140 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:51.015251 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:51.015760 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.016742 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.018545 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.019529 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.019640 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.019710 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:51.020094 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 21:36:51.020271 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:36:51.021283 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.015024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50911 (local address 127.0.97.2:36219)
0926 21:36:51.015260 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:51.015266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:51.015577 (+ 311us) server_negotiation.cc:407] Connection header received
0926 21:36:51.015716 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:51.015720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:51.015797 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:51.015861 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:51.016376 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.017242 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.019022 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.020068 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.020177 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.020267 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:51.020826 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 21:36:51.020966 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:36:51.021306 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e9b0bf552324d76a057d598dcb383a9" candidate_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff87a116bbd424c896d96c50e4e008d" is_pre_election: true
I20260926 21:36:51.021601 2286 raft_consensus.cc:2454] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d6d31ed2fa4e8793b81619badaf61c in term 0.
I20260926 21:36:51.022121 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.014902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36219 (local address 127.0.97.3:50911)
0926 21:36:51.015481 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:51.015495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:51.015595 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:51.015910 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:51.015914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:51.015937 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:51.016242 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.016252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.017365 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.017369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.018842 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.018855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.020351 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.020358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.020567 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.020608 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 21:36:51.021471 (+ 863us) client_negotiation.cc:240] Negotiation successful
0926 21:36:51.021636 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":24,"thread_start_us":328,"threads_started":1}
I20260926 21:36:51.022152 2498 leader_election.cc:302] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ff87a116bbd424c896d96c50e4e008d, b6d6d31ed2fa4e8793b81619badaf61c; no voters:
I20260926 21:36:51.022594 2640 raft_consensus.cc:2781] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:36:51.022749 2640 raft_consensus.cc:482] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:36:51.022861 2640 raft_consensus.cc:3037] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:51.023371 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e9b0bf552324d76a057d598dcb383a9" candidate_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" is_pre_election: true
I20260926 21:36:51.023837 2425 raft_consensus.cc:2454] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d6d31ed2fa4e8793b81619badaf61c in term 0.
I20260926 21:36:51.025653 2640 raft_consensus.cc:504] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.026186 2640 leader_election.cc:288] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [CANDIDATE]: Term 1 election: Requested vote from peers 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187), 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219)
I20260926 21:36:51.026597 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e9b0bf552324d76a057d598dcb383a9" candidate_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff87a116bbd424c896d96c50e4e008d"
I20260926 21:36:51.026724 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e9b0bf552324d76a057d598dcb383a9" candidate_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1bb9b74bbd4acca5da79884c7d9273"
I20260926 21:36:51.026789 2286 raft_consensus.cc:3037] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:51.026942 2425 raft_consensus.cc:3037] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:51.028744 2286 raft_consensus.cc:2454] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d6d31ed2fa4e8793b81619badaf61c in term 1.
I20260926 21:36:51.029122 2498 leader_election.cc:302] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ff87a116bbd424c896d96c50e4e008d, b6d6d31ed2fa4e8793b81619badaf61c; no voters:
I20260926 21:36:51.029240 2425 raft_consensus.cc:2454] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d6d31ed2fa4e8793b81619badaf61c in term 1.
I20260926 21:36:51.029472 2640 raft_consensus.cc:2781] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:51.029969 2640 raft_consensus.cc:686] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [term 1 LEADER]: Becoming Leader. State: Replica: b6d6d31ed2fa4e8793b81619badaf61c, State: Running, Role: LEADER
I20260926 21:36:51.030493 2640 consensus_queue.cc:237] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.033942 2656 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: TxnStatusTablet state changed. Reason: New leader b6d6d31ed2fa4e8793b81619badaf61c. Latest consensus state: current_term: 1 leader_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:51.034336 2656 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:36:51.035030 2658 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 21:36:51.035307 2658 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 21:36:51.035279 2128 catalog_manager.cc:5495] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c reported cstate change: term changed from 0 to 1, leader changed from <none> to b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } health_report { overall_health: HEALTHY } } }
I20260926 21:36:51.060913 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.053534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44527 (local address 127.0.0.1:46208)
0926 21:36:51.053787 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:51.053810 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:51.053952 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:51.054674 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:51.054679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:51.054692 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:36:51.054887 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.054896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.056370 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.056375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.057972 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.057985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.058974 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.058981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.059197 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.059266 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:36:51.060321 (+ 1055us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:36:51.060356 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:36:51.060436 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 21:36:51.060594 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 21:36:51.061568 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.053535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.0.97.3:44527)
0926 21:36:51.054233 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:51.054239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:51.054257 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:36:51.054357 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:51.054362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:51.054485 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:51.054647 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:51.055036 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.056228 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.058122 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.058780 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.058919 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.059459 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:51.060126 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:51.060265 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:51.061130 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 21:36:51.061259 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576}
I20260926 21:36:51.069550 2425 raft_consensus.cc:1261] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Refusing update from remote peer b6d6d31ed2fa4e8793b81619badaf61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:51.069679 2286 raft_consensus.cc:1261] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Refusing update from remote peer b6d6d31ed2fa4e8793b81619badaf61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:51.070423 2640 consensus_queue.cc:1035] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:51.070729 2656 consensus_queue.cc:1035] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:51.079329 2639 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: TxnStatusTablet state changed. Reason: New leader b6d6d31ed2fa4e8793b81619badaf61c. Latest consensus state: current_term: 1 leader_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:51.079645 2639 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:51.087333 2661 mvcc.cc:205] Tried to move back new op lower bound from 7333718470931333120 to 7333718470785171456. Current Snapshot: MvccSnapshot[applied={T|T < 7333718470931333120}]
I20260926 21:36:51.089787 2640 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:51.090067 2640 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:36:51.092973 2639 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:51.093271 2639 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P 0e1bb9b74bbd4acca5da79884c7d9273: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:51.096493 2641 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: TxnStatusTablet state changed. Reason: New leader b6d6d31ed2fa4e8793b81619badaf61c. Latest consensus state: current_term: 1 leader_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:51.096781 2641 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:51.101588 2656 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:51.101862 2656 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P b6d6d31ed2fa4e8793b81619badaf61c: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:36:51.102471 2641 tablet_replica.cc:439] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } } }
I20260926 21:36:51.102726 2641 tablet_replica.cc:442] T 4e9b0bf552324d76a057d598dcb383a9 P 9ff87a116bbd424c896d96c50e4e008d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:36:51.105927 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.099032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.0.97.2:36219)
0926 21:36:51.099221 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:51.099226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:51.099242 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:36:51.099321 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:51.099325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:51.099449 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:51.099595 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:51.099983 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.101013 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.103135 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.103796 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.103930 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.104565 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:51.105296 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:51.105458 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:51.105544 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 21:36:51.105630 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 21:36:51.122264 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.116472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48111 (local address 127.0.97.3:44527)
0926 21:36:51.116658 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:51.116667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:51.117156 (+ 489us) server_negotiation.cc:407] Connection header received
0926 21:36:51.117444 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:51.117448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:51.117510 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:51.117621 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:51.118111 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.118829 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.120314 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.120814 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.120943 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.121508 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:51.121602 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:51.121712 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:51.121809 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 21:36:51.121889 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:36:51.122321 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.116395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44527 (local address 127.0.97.2:48111)
0926 21:36:51.117019 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:51.117034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:51.117286 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:51.117636 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:51.117641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:51.117652 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:36:51.117970 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.117978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.118980 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.118986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.120173 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.120185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.120995 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.121003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.121173 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.121277 (+ 104us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:36:51.121708 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:36:51.121718 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:36:51.121793 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 21:36:51.121928 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":352,"threads_started":1}
I20260926 21:36:51.131457 2286 raft_consensus.cc:1261] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Refusing update from remote peer 0e1bb9b74bbd4acca5da79884c7d9273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:51.131997 2569 raft_consensus.cc:1261] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Refusing update from remote peer 0e1bb9b74bbd4acca5da79884c7d9273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:51.132751 2639 consensus_queue.cc:1035] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:51.133040 2644 consensus_queue.cc:1035] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:51.139776 2662 mvcc.cc:205] Tried to move back new op lower bound from 7333718471188590592 to 7333718469797543936. Current Snapshot: MvccSnapshot[applied={T|T < 7333718471188590592}]
I20260926 21:36:51.173511 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.166158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42575 (local address 127.0.97.2:36219)
0926 21:36:51.166346 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:51.166352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:51.167092 (+ 740us) server_negotiation.cc:407] Connection header received
0926 21:36:51.167362 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:51.167368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:51.167446 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:51.167553 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:51.168267 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.169390 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.171432 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.172103 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.172224 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.172676 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:51.172806 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:51.172909 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:51.173084 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 21:36:51.173166 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 21:36:51.173600 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:51.166226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36219 (local address 127.0.97.3:42575)
0926 21:36:51.166997 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:51.167018 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:51.167206 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:51.167591 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:51.167597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:51.167612 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:36:51.168046 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.168059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.169527 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.169531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.171109 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:51.171123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:51.172304 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:51.172309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:51.172441 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:51.172498 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:36:51.172945 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:36:51.172962 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 21:36:51.173049 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 21:36:51.173211 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":382,"threads_started":1}
I20260926 21:36:51.213366 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:51.213682 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:51.217571 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7: Sending DeleteTablet for 3 replicas of tablet 93a0d15e6e7d4aadbb9012c82aaef6e9
I20260926 21:36:51.218632 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a0d15e6e7d4aadbb9012c82aaef6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46200
I20260926 21:36:51.219493 2682 tablet_replica.cc:330] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: stopping tablet replica
I20260926 21:36:51.219549 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a0d15e6e7d4aadbb9012c82aaef6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37358
I20260926 21:36:51.220113 2682 raft_consensus.cc:2229] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:51.220486 2682 raft_consensus.cc:2258] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:51.220430 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 93a0d15e6e7d4aadbb9012c82aaef6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33394
I20260926 21:36:51.222484 2682 ts_tablet_manager.cc:1905] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:51.228061 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:36:51.228745 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:36:51.230580 2682 ts_tablet_manager.cc:1918] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:36:51.230827 2682 log.cc:1196] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/93a0d15e6e7d4aadbb9012c82aaef6e9
I20260926 21:36:51.231526 2682 ts_tablet_manager.cc:1939] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P b6d6d31ed2fa4e8793b81619badaf61c: Deleting consensus metadata
I20260926 21:36:51.234654 2114 catalog_manager.cc:4841] TS b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527): tablet 93a0d15e6e7d4aadbb9012c82aaef6e9 (table test-table [id=a9bbb3a6421c4ff89b36caaf530b06bd]) successfully deleted
I20260926 21:36:51.236817 2265 tablet_service.cc:1467] Processing CreateTablet for tablet a9b2b379b38740abb14119a6ddfb0f8a (DEFAULT_TABLE table=test-table [id=17dadb2ff40245a7a39c74408ddcffe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:51.237001 2548 tablet_service.cc:1467] Processing CreateTablet for tablet a9b2b379b38740abb14119a6ddfb0f8a (DEFAULT_TABLE table=test-table [id=17dadb2ff40245a7a39c74408ddcffe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:51.237018 2404 tablet_service.cc:1467] Processing CreateTablet for tablet a9b2b379b38740abb14119a6ddfb0f8a (DEFAULT_TABLE table=test-table [id=17dadb2ff40245a7a39c74408ddcffe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:51.237382 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9b2b379b38740abb14119a6ddfb0f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:51.237458 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9b2b379b38740abb14119a6ddfb0f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:51.237533 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9b2b379b38740abb14119a6ddfb0f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:51.242055 2635 tablet_bootstrap.cc:492] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: Bootstrap starting.
I20260926 21:36:51.243745 2635 tablet_bootstrap.cc:654] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:51.244095 2634 tablet_bootstrap.cc:492] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: Bootstrap starting.
I20260926 21:36:51.246142 2634 tablet_bootstrap.cc:654] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:51.253537 2635 tablet_bootstrap.cc:492] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: No bootstrap required, opened a new log
I20260926 21:36:51.253854 2635 ts_tablet_manager.cc:1397] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s
I20260926 21:36:51.254729 2635 raft_consensus.cc:348] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.255103 2635 raft_consensus.cc:374] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:51.255223 2635 raft_consensus.cc:729] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff87a116bbd424c896d96c50e4e008d, State: Initialized, Role: FOLLOWER
I20260926 21:36:51.255576 2635 consensus_queue.cc:260] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.256891 2635 ts_tablet_manager.cc:1428] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 21:36:51.258559 2634 tablet_bootstrap.cc:492] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: No bootstrap required, opened a new log
I20260926 21:36:51.258740 2634 ts_tablet_manager.cc:1397] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20260926 21:36:51.259578 2634 raft_consensus.cc:348] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.259822 2634 raft_consensus.cc:374] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:51.259898 2634 raft_consensus.cc:729] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1bb9b74bbd4acca5da79884c7d9273, State: Initialized, Role: FOLLOWER
I20260926 21:36:51.260170 2634 consensus_queue.cc:260] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.260756 2634 ts_tablet_manager.cc:1428] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 21:36:51.265548 2633 tablet_bootstrap.cc:492] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: Bootstrap starting.
I20260926 21:36:51.266181 2683 tablet_replica.cc:330] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: stopping tablet replica
I20260926 21:36:51.266746 2684 tablet_replica.cc:330] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: stopping tablet replica
I20260926 21:36:51.267274 2684 raft_consensus.cc:2229] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:36:51.267444 2633 tablet_bootstrap.cc:654] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:51.267834 2683 raft_consensus.cc:2229] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:51.268266 2683 raft_consensus.cc:2258] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:51.268953 2684 raft_consensus.cc:2258] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:51.271458 2683 ts_tablet_manager.cc:1905] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:51.273587 2684 ts_tablet_manager.cc:1905] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:51.278714 2633 tablet_bootstrap.cc:492] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: No bootstrap required, opened a new log
I20260926 21:36:51.278920 2633 ts_tablet_manager.cc:1397] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260926 21:36:51.279780 2633 raft_consensus.cc:348] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.280052 2633 raft_consensus.cc:374] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:51.280138 2633 raft_consensus.cc:729] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d6d31ed2fa4e8793b81619badaf61c, State: Initialized, Role: FOLLOWER
I20260926 21:36:51.280424 2633 consensus_queue.cc:260] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.283486 2633 ts_tablet_manager.cc:1428] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260926 21:36:51.288352 2683 ts_tablet_manager.cc:1918] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:36:51.288530 2683 log.cc:1196] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/93a0d15e6e7d4aadbb9012c82aaef6e9
I20260926 21:36:51.289076 2683 ts_tablet_manager.cc:1939] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 9ff87a116bbd424c896d96c50e4e008d: Deleting consensus metadata
I20260926 21:36:51.291023 2684 ts_tablet_manager.cc:1918] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:36:51.291221 2684 log.cc:1196] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/93a0d15e6e7d4aadbb9012c82aaef6e9
I20260926 21:36:51.291785 2684 ts_tablet_manager.cc:1939] T 93a0d15e6e7d4aadbb9012c82aaef6e9 P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting consensus metadata
I20260926 21:36:51.293542 2116 catalog_manager.cc:4841] TS 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219): tablet 93a0d15e6e7d4aadbb9012c82aaef6e9 (table test-table [id=a9bbb3a6421c4ff89b36caaf530b06bd]) successfully deleted
I20260926 21:36:51.294580 2116 catalog_manager.cc:4841] TS 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187): tablet 93a0d15e6e7d4aadbb9012c82aaef6e9 (table test-table [id=a9bbb3a6421c4ff89b36caaf530b06bd]) successfully deleted
I20260926 21:36:51.371717 2639 raft_consensus.cc:482] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:36:51.371968 2639 raft_consensus.cc:504] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.372804 2639 leader_election.cc:288] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187), b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527)
I20260926 21:36:51.373515 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9b2b379b38740abb14119a6ddfb0f8a" candidate_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff87a116bbd424c896d96c50e4e008d" is_pre_election: true
I20260926 21:36:51.373572 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9b2b379b38740abb14119a6ddfb0f8a" candidate_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" is_pre_election: true
I20260926 21:36:51.373775 2286 raft_consensus.cc:2454] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1bb9b74bbd4acca5da79884c7d9273 in term 0.
I20260926 21:36:51.373818 2569 raft_consensus.cc:2454] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1bb9b74bbd4acca5da79884c7d9273 in term 0.
I20260926 21:36:51.374325 2357 leader_election.cc:302] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1bb9b74bbd4acca5da79884c7d9273, b6d6d31ed2fa4e8793b81619badaf61c; no voters:
I20260926 21:36:51.374645 2639 raft_consensus.cc:2781] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:36:51.374773 2639 raft_consensus.cc:482] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:36:51.374864 2639 raft_consensus.cc:3037] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:51.378008 2639 raft_consensus.cc:504] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.379166 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9b2b379b38740abb14119a6ddfb0f8a" candidate_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff87a116bbd424c896d96c50e4e008d"
I20260926 21:36:51.379218 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9b2b379b38740abb14119a6ddfb0f8a" candidate_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d6d31ed2fa4e8793b81619badaf61c"
I20260926 21:36:51.379379 2286 raft_consensus.cc:3037] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:51.379424 2569 raft_consensus.cc:3037] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:51.382393 2286 raft_consensus.cc:2454] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1bb9b74bbd4acca5da79884c7d9273 in term 1.
I20260926 21:36:51.382583 2569 raft_consensus.cc:2454] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1bb9b74bbd4acca5da79884c7d9273 in term 1.
I20260926 21:36:51.383054 2359 leader_election.cc:302] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e1bb9b74bbd4acca5da79884c7d9273, 9ff87a116bbd424c896d96c50e4e008d, b6d6d31ed2fa4e8793b81619badaf61c; no voters:
I20260926 21:36:51.383376 2644 raft_consensus.cc:2781] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:51.383536 2644 raft_consensus.cc:686] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 LEADER]: Becoming Leader. State: Replica: 0e1bb9b74bbd4acca5da79884c7d9273, State: Running, Role: LEADER
I20260926 21:36:51.383781 2644 consensus_queue.cc:237] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:51.383209 2639 leader_election.cc:288] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [CANDIDATE]: Term 1 election: Requested vote from peers 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187), b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527)
I20260926 21:36:51.386902 2129 catalog_manager.cc:5495] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260926 21:36:51.433697 2286 raft_consensus.cc:1261] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Refusing update from remote peer 0e1bb9b74bbd4acca5da79884c7d9273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:51.435015 2569 raft_consensus.cc:1261] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Refusing update from remote peer 0e1bb9b74bbd4acca5da79884c7d9273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:51.435359 2644 consensus_queue.cc:1035] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:51.435835 2644 consensus_queue.cc:1035] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 21:36:51.464582 2662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 21:36:53.088500 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:53.088758 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:53.092572 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7: Sending DeleteTablet for 3 replicas of tablet a9b2b379b38740abb14119a6ddfb0f8a
I20260926 21:36:53.093442 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet a9b2b379b38740abb14119a6ddfb0f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37358
I20260926 21:36:53.093463 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet a9b2b379b38740abb14119a6ddfb0f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33394
I20260926 21:36:53.093461 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet a9b2b379b38740abb14119a6ddfb0f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46200
I20260926 21:36:53.105046 2712 tablet_replica.cc:330] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: stopping tablet replica
I20260926 21:36:53.105481 2712 raft_consensus.cc:2229] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:53.105796 2712 raft_consensus.cc:2258] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:53.107275 2713 tablet_replica.cc:330] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: stopping tablet replica
I20260926 21:36:53.107455 2714 tablet_replica.cc:330] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: stopping tablet replica
I20260926 21:36:53.107599 2713 raft_consensus.cc:2229] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:36:53.107645 2714 raft_consensus.cc:2229] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:53.107681 2712 ts_tablet_manager.cc:1905] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:53.107873 2714 raft_consensus.cc:2258] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:53.108479 2713 raft_consensus.cc:2258] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:53.108762 2714 ts_tablet_manager.cc:1905] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:53.109448 2713 ts_tablet_manager.cc:1905] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:53.115219 2714 ts_tablet_manager.cc:1918] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:36:53.115243 2712 ts_tablet_manager.cc:1918] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:36:53.115362 2712 log.cc:1196] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/a9b2b379b38740abb14119a6ddfb0f8a
I20260926 21:36:53.115386 2714 log.cc:1196] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/a9b2b379b38740abb14119a6ddfb0f8a
I20260926 21:36:53.115818 2714 ts_tablet_manager.cc:1939] T a9b2b379b38740abb14119a6ddfb0f8a P 9ff87a116bbd424c896d96c50e4e008d: Deleting consensus metadata
I20260926 21:36:53.115819 2712 ts_tablet_manager.cc:1939] T a9b2b379b38740abb14119a6ddfb0f8a P b6d6d31ed2fa4e8793b81619badaf61c: Deleting consensus metadata
I20260926 21:36:53.116077 2713 ts_tablet_manager.cc:1918] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:36:53.116256 2713 log.cc:1196] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/a9b2b379b38740abb14119a6ddfb0f8a
I20260926 21:36:53.116695 2713 ts_tablet_manager.cc:1939] T a9b2b379b38740abb14119a6ddfb0f8a P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting consensus metadata
I20260926 21:36:53.116997 2116 catalog_manager.cc:4841] TS 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187): tablet a9b2b379b38740abb14119a6ddfb0f8a (table test-table [id=17dadb2ff40245a7a39c74408ddcffe2]) successfully deleted
I20260926 21:36:53.117025 2114 catalog_manager.cc:4841] TS b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527): tablet a9b2b379b38740abb14119a6ddfb0f8a (table test-table [id=17dadb2ff40245a7a39c74408ddcffe2]) successfully deleted
I20260926 21:36:53.118098 2116 catalog_manager.cc:4841] TS 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219): tablet a9b2b379b38740abb14119a6ddfb0f8a (table test-table [id=17dadb2ff40245a7a39c74408ddcffe2]) successfully deleted
Sep 26 21:36:53 dist-test-slave-lhhq krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458613, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:36:53 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458613, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:53.151330 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.137115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.0.97.62:34987)
0926 21:36:53.137749 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:53.137755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:53.137780 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:36:53.137994 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:53.137998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:53.138058 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:53.138171 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:53.139840 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.140590 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.141902 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.142320 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.142461 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.146248 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:53.146253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:53.146256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:53.146286 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:53.148762 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:53.149458 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:53.149466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:53.149474 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:53.149631 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:53.149996 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:53.150000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:53.150004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:53.150255 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:53.150495 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:53.150793 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 21:36:53.151014 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":279,"threads_started":1}
I20260926 21:36:53.154814 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:36:53.155401 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:36:53.161842 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 793b6648b3744b1f989c93f4685d961d (DEFAULT_TABLE table=test-table [id=e97c3248d2c14c40ae14ce953f7592e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:53.161979 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 793b6648b3744b1f989c93f4685d961d (DEFAULT_TABLE table=test-table [id=e97c3248d2c14c40ae14ce953f7592e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:53.162189 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 793b6648b3744b1f989c93f4685d961d (DEFAULT_TABLE table=test-table [id=e97c3248d2c14c40ae14ce953f7592e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:53.162303 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793b6648b3744b1f989c93f4685d961d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:53.162386 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793b6648b3744b1f989c93f4685d961d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:53.162564 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793b6648b3744b1f989c93f4685d961d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:53.168643 2723 tablet_bootstrap.cc:492] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: Bootstrap starting.
I20260926 21:36:53.169617 2725 tablet_bootstrap.cc:492] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: Bootstrap starting.
I20260926 21:36:53.170424 2724 tablet_bootstrap.cc:492] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: Bootstrap starting.
I20260926 21:36:53.170778 2723 tablet_bootstrap.cc:654] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:53.171401 2725 tablet_bootstrap.cc:654] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:53.172049 2724 tablet_bootstrap.cc:654] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:53.174489 2724 tablet_bootstrap.cc:492] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: No bootstrap required, opened a new log
I20260926 21:36:53.174631 2724 ts_tablet_manager.cc:1397] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 21:36:53.174671 2723 tablet_bootstrap.cc:492] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: No bootstrap required, opened a new log
I20260926 21:36:53.174671 2725 tablet_bootstrap.cc:492] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: No bootstrap required, opened a new log
I20260926 21:36:53.174796 2723 ts_tablet_manager.cc:1397] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 21:36:53.174825 2725 ts_tablet_manager.cc:1397] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 21:36:53.175421 2724 raft_consensus.cc:348] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.175647 2724 raft_consensus.cc:374] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:53.175546 2725 raft_consensus.cc:348] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.175797 2725 raft_consensus.cc:374] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:53.175791 2724 raft_consensus.cc:729] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1bb9b74bbd4acca5da79884c7d9273, State: Initialized, Role: FOLLOWER
I20260926 21:36:53.175889 2725 raft_consensus.cc:729] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d6d31ed2fa4e8793b81619badaf61c, State: Initialized, Role: FOLLOWER
I20260926 21:36:53.176018 2723 raft_consensus.cc:348] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.176074 2724 consensus_queue.cc:260] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.176100 2725 consensus_queue.cc:260] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.176236 2723 raft_consensus.cc:374] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:53.176334 2723 raft_consensus.cc:729] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff87a116bbd424c896d96c50e4e008d, State: Initialized, Role: FOLLOWER
I20260926 21:36:53.176599 2724 ts_tablet_manager.cc:1428] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 21:36:53.176626 2725 ts_tablet_manager.cc:1428] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 21:36:53.176558 2723 consensus_queue.cc:260] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.177496 2723 ts_tablet_manager.cc:1428] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 21:36:53.380548 2729 raft_consensus.cc:482] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:36:53.381270 2729 raft_consensus.cc:504] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.382776 2729 leader_election.cc:288] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219), b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527)
I20260926 21:36:53.403013 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.383236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55495 (local address 127.0.97.3:44527)
0926 21:36:53.396387 (+ 13151us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:53.396394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:53.396420 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:36:53.396501 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:53.396509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:53.396583 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:53.396706 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:53.397340 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.398327 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.400523 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.401486 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.401584 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.401672 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:53.402045 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 21:36:53.402191 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13001,"thread_start_us":12911,"threads_started":1}
I20260926 21:36:53.403029 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.383235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39187 (local address 127.0.97.2:36219)
0926 21:36:53.396173 (+ 12938us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:53.396179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:53.396843 (+ 664us) server_negotiation.cc:407] Connection header received
0926 21:36:53.397047 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:53.397052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:53.397134 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:53.397279 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:36:53.397919 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.398827 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.400880 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.401835 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.401986 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.402113 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:53.402532 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 21:36:53.402691 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12775,"thread_start_us":12610,"threads_started":1}
I20260926 21:36:53.403067 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.383206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44527 (local address 127.0.97.1:55495)
0926 21:36:53.395752 (+ 12546us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:53.395787 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:53.396008 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:53.396739 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:53.396743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:53.396766 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:53.397164 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:53.397181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.398487 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.398494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:53.400362 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:53.400375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.401673 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.401680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:53.401862 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.401910 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 21:36:53.402002 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 21:36:53.402168 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12334,"spinlock_wait_cycles":14080,"thread_start_us":13158,"threads_started":1}
I20260926 21:36:53.403656 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.383234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36219 (local address 127.0.97.1:39187)
0926 21:36:53.396718 (+ 13484us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:53.396736 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:53.396897 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:53.397295 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:53.397301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:53.397326 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:36:53.397731 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:53.397743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.398971 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.398976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:53.400724 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:53.400740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.402097 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.402104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:53.402320 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.402358 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 21:36:53.402463 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 21:36:53.403015 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13323,"spinlock_wait_cycles":22656,"thread_start_us":12084,"threads_started":1}
I20260926 21:36:53.404066 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793b6648b3744b1f989c93f4685d961d" candidate_uuid: "9ff87a116bbd424c896d96c50e4e008d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" is_pre_election: true
I20260926 21:36:53.404343 2569 raft_consensus.cc:2454] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff87a116bbd424c896d96c50e4e008d in term 0.
I20260926 21:36:53.404315 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793b6648b3744b1f989c93f4685d961d" candidate_uuid: "9ff87a116bbd424c896d96c50e4e008d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" is_pre_election: true
I20260926 21:36:53.404537 2425 raft_consensus.cc:2454] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff87a116bbd424c896d96c50e4e008d in term 0.
I20260926 21:36:53.404922 2217 leader_election.cc:302] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ff87a116bbd424c896d96c50e4e008d, b6d6d31ed2fa4e8793b81619badaf61c; no voters:
I20260926 21:36:53.405462 2729 raft_consensus.cc:2781] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:36:53.405588 2729 raft_consensus.cc:482] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:36:53.405687 2729 raft_consensus.cc:3037] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:53.407847 2729 raft_consensus.cc:504] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.408463 2729 leader_election.cc:288] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [CANDIDATE]: Term 1 election: Requested vote from peers 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219), b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527)
I20260926 21:36:53.408949 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793b6648b3744b1f989c93f4685d961d" candidate_uuid: "9ff87a116bbd424c896d96c50e4e008d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1bb9b74bbd4acca5da79884c7d9273"
I20260926 21:36:53.409140 2425 raft_consensus.cc:3037] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:53.409024 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793b6648b3744b1f989c93f4685d961d" candidate_uuid: "9ff87a116bbd424c896d96c50e4e008d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d6d31ed2fa4e8793b81619badaf61c"
I20260926 21:36:53.409317 2569 raft_consensus.cc:3037] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:53.411751 2569 raft_consensus.cc:2454] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff87a116bbd424c896d96c50e4e008d in term 1.
I20260926 21:36:53.411826 2425 raft_consensus.cc:2454] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff87a116bbd424c896d96c50e4e008d in term 1.
I20260926 21:36:53.412225 2217 leader_election.cc:302] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ff87a116bbd424c896d96c50e4e008d, b6d6d31ed2fa4e8793b81619badaf61c; no voters:
I20260926 21:36:53.412539 2729 raft_consensus.cc:2781] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:53.412990 2729 raft_consensus.cc:686] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 1 LEADER]: Becoming Leader. State: Replica: 9ff87a116bbd424c896d96c50e4e008d, State: Running, Role: LEADER
I20260926 21:36:53.413403 2729 consensus_queue.cc:237] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.416774 2128 catalog_manager.cc:5495] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9ff87a116bbd424c896d96c50e4e008d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260926 21:36:53.475987 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.470276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.0.97.1:36187)
0926 21:36:53.470779 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:53.470784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:53.470802 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:36:53.470885 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:53.470888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:53.470967 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:53.471106 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:53.471430 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.472295 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.473659 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.474055 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.474146 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.474615 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:53.475281 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:53.475453 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:53.475532 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 21:36:53.475615 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":276,"threads_started":1}
I20260926 21:36:53.491878 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.486001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46427 (local address 127.0.97.3:44527)
0926 21:36:53.486249 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:53.486259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:53.486663 (+ 404us) server_negotiation.cc:407] Connection header received
0926 21:36:53.486899 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:53.486912 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:53.486980 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:53.487075 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:53.487638 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.488328 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.490166 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.490596 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.490734 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.491164 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:53.491268 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:53.491368 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:53.491509 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 21:36:53.491585 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:36:53.492002 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.485907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44527 (local address 127.0.97.1:46427)
0926 21:36:53.486557 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:53.486575 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:53.486762 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:53.487104 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:53.487133 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:53.487146 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:36:53.487490 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:53.487499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.488462 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.488470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:53.490011 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:53.490023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.490785 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.490793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:53.490926 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.490988 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:36:53.491373 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:36:53.491387 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 21:36:53.491486 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 21:36:53.491622 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":350,"threads_started":1}
I20260926 21:36:53.499907 2425 raft_consensus.cc:1261] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Refusing update from remote peer 9ff87a116bbd424c896d96c50e4e008d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:53.499956 2569 raft_consensus.cc:1261] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Refusing update from remote peer 9ff87a116bbd424c896d96c50e4e008d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:53.500643 2734 consensus_queue.cc:1035] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:53.500967 2729 consensus_queue.cc:1035] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:53.504930 2695 mvcc.cc:205] Tried to move back new op lower bound from 7333718480889720832 to 7333718480544727040. Current Snapshot: MvccSnapshot[applied={T|T < 7333718480889720832}]
I20260926 21:36:53.525943 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.520475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59085 (local address 127.0.97.1:36187)
0926 21:36:53.520673 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:53.520679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:53.521002 (+ 323us) server_negotiation.cc:407] Connection header received
0926 21:36:53.521134 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:53.521137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:53.521240 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:53.521325 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:36:53.521888 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.522725 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.524512 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.524899 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.524997 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.525387 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:36:53.525492 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:36:53.525558 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:53.525673 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 21:36:53.525730 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:36:53.526036 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:53.520323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36187 (local address 127.0.97.3:59085)
0926 21:36:53.520929 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:53.520945 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:53.521050 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:53.521394 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:53.521398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:53.521407 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:36:53.521594 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:53.521602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.522867 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.522872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:53.524351 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:53.524365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:53.525033 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:53.525037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:53.525159 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:53.525243 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:36:53.525572 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:36:53.525580 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 21:36:53.525637 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 21:36:53.525737 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":315,"threads_started":1}
I20260926 21:36:53.556382 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:53.556660 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:53.560648 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7: Sending DeleteTablet for 3 replicas of tablet 793b6648b3744b1f989c93f4685d961d
I20260926 21:36:53.561626 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 793b6648b3744b1f989c93f4685d961d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37358
I20260926 21:36:53.561765 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 793b6648b3744b1f989c93f4685d961d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33394
I20260926 21:36:53.561934 2714 tablet_replica.cc:330] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: stopping tablet replica
I20260926 21:36:53.562036 2713 tablet_replica.cc:330] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: stopping tablet replica
I20260926 21:36:53.562139 2714 raft_consensus.cc:2229] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:36:53.562198 2713 raft_consensus.cc:2229] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:53.562357 2713 raft_consensus.cc:2258] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:53.562638 2714 raft_consensus.cc:2258] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:53.563494 2713 ts_tablet_manager.cc:1905] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:53.564306 2714 ts_tablet_manager.cc:1905] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:53.568495 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 793b6648b3744b1f989c93f4685d961d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46200
I20260926 21:36:53.569048 2712 tablet_replica.cc:330] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: stopping tablet replica
I20260926 21:36:53.569267 2712 raft_consensus.cc:2229] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:53.569475 2712 raft_consensus.cc:2258] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:53.572309 2714 ts_tablet_manager.cc:1918] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:36:53.572621 2714 log.cc:1196] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/793b6648b3744b1f989c93f4685d961d
I20260926 21:36:53.573977 2712 ts_tablet_manager.cc:1905] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:53.579694 2713 ts_tablet_manager.cc:1918] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:36:53.579854 2713 log.cc:1196] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/793b6648b3744b1f989c93f4685d961d
I20260926 21:36:53.580253 2713 ts_tablet_manager.cc:1939] T 793b6648b3744b1f989c93f4685d961d P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting consensus metadata
I20260926 21:36:53.581292 2116 catalog_manager.cc:4841] TS 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219): tablet 793b6648b3744b1f989c93f4685d961d (table test-table [id=e97c3248d2c14c40ae14ce953f7592e6]) successfully deleted
I20260926 21:36:53.581315 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:36:53.582381 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:36:53.585592 2712 ts_tablet_manager.cc:1918] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:36:53.585755 2712 log.cc:1196] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/793b6648b3744b1f989c93f4685d961d
I20260926 21:36:53.586573 2712 ts_tablet_manager.cc:1939] T 793b6648b3744b1f989c93f4685d961d P b6d6d31ed2fa4e8793b81619badaf61c: Deleting consensus metadata
I20260926 21:36:53.587757 2114 catalog_manager.cc:4841] TS b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527): tablet 793b6648b3744b1f989c93f4685d961d (table test-table [id=e97c3248d2c14c40ae14ce953f7592e6]) successfully deleted
I20260926 21:36:53.591920 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0d5b74bea04f4f9545b3668f0faa2d (DEFAULT_TABLE table=test-table [id=8ec2ee7cd8fc4d7ebb77f839091135b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:53.592041 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0d5b74bea04f4f9545b3668f0faa2d (DEFAULT_TABLE table=test-table [id=8ec2ee7cd8fc4d7ebb77f839091135b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:53.592212 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0d5b74bea04f4f9545b3668f0faa2d (DEFAULT_TABLE table=test-table [id=8ec2ee7cd8fc4d7ebb77f839091135b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:36:53.592360 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0d5b74bea04f4f9545b3668f0faa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:53.592489 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0d5b74bea04f4f9545b3668f0faa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:53.592840 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0d5b74bea04f4f9545b3668f0faa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:53.598862 2724 tablet_bootstrap.cc:492] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: Bootstrap starting.
I20260926 21:36:53.599287 2723 tablet_bootstrap.cc:492] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: Bootstrap starting.
I20260926 21:36:53.599813 2714 ts_tablet_manager.cc:1939] T 793b6648b3744b1f989c93f4685d961d P 9ff87a116bbd424c896d96c50e4e008d: Deleting consensus metadata
I20260926 21:36:53.600654 2724 tablet_bootstrap.cc:654] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:53.601369 2723 tablet_bootstrap.cc:654] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:53.602586 2116 catalog_manager.cc:4841] TS 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187): tablet 793b6648b3744b1f989c93f4685d961d (table test-table [id=e97c3248d2c14c40ae14ce953f7592e6]) successfully deleted
I20260926 21:36:53.604934 2725 tablet_bootstrap.cc:492] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: Bootstrap starting.
I20260926 21:36:53.606767 2725 tablet_bootstrap.cc:654] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:53.611476 2724 tablet_bootstrap.cc:492] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: No bootstrap required, opened a new log
I20260926 21:36:53.611702 2724 ts_tablet_manager.cc:1397] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260926 21:36:53.612448 2724 raft_consensus.cc:348] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.613317 2724 raft_consensus.cc:374] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:53.613438 2724 raft_consensus.cc:729] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1bb9b74bbd4acca5da79884c7d9273, State: Initialized, Role: FOLLOWER
I20260926 21:36:53.613742 2724 consensus_queue.cc:260] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.615255 2724 ts_tablet_manager.cc:1428] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 21:36:53.616961 2723 tablet_bootstrap.cc:492] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: No bootstrap required, opened a new log
I20260926 21:36:53.617158 2723 ts_tablet_manager.cc:1397] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20260926 21:36:53.617828 2723 raft_consensus.cc:348] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.618048 2723 raft_consensus.cc:374] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:53.618146 2723 raft_consensus.cc:729] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff87a116bbd424c896d96c50e4e008d, State: Initialized, Role: FOLLOWER
I20260926 21:36:53.618436 2723 consensus_queue.cc:260] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.619082 2723 ts_tablet_manager.cc:1428] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 21:36:53.621230 2725 tablet_bootstrap.cc:492] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: No bootstrap required, opened a new log
I20260926 21:36:53.621399 2725 ts_tablet_manager.cc:1397] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.001s
I20260926 21:36:53.622632 2725 raft_consensus.cc:348] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.622888 2725 raft_consensus.cc:374] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:53.622972 2725 raft_consensus.cc:729] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d6d31ed2fa4e8793b81619badaf61c, State: Initialized, Role: FOLLOWER
I20260926 21:36:53.623209 2725 consensus_queue.cc:260] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.624137 2725 ts_tablet_manager.cc:1428] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 21:36:53.840539 2734 raft_consensus.cc:482] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:36:53.840806 2734 raft_consensus.cc:504] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.841634 2734 leader_election.cc:288] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219), b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527)
I20260926 21:36:53.842160 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0d5b74bea04f4f9545b3668f0faa2d" candidate_uuid: "9ff87a116bbd424c896d96c50e4e008d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" is_pre_election: true
I20260926 21:36:53.842275 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0d5b74bea04f4f9545b3668f0faa2d" candidate_uuid: "9ff87a116bbd424c896d96c50e4e008d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" is_pre_election: true
I20260926 21:36:53.842393 2425 raft_consensus.cc:2454] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff87a116bbd424c896d96c50e4e008d in term 0.
I20260926 21:36:53.842535 2569 raft_consensus.cc:2454] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff87a116bbd424c896d96c50e4e008d in term 0.
I20260926 21:36:53.842866 2219 leader_election.cc:302] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1bb9b74bbd4acca5da79884c7d9273, 9ff87a116bbd424c896d96c50e4e008d; no voters:
I20260926 21:36:53.843217 2734 raft_consensus.cc:2781] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:36:53.843325 2734 raft_consensus.cc:482] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:36:53.843461 2734 raft_consensus.cc:3037] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:53.845917 2734 raft_consensus.cc:504] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.846539 2734 leader_election.cc:288] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [CANDIDATE]: Term 1 election: Requested vote from peers 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219), b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527)
I20260926 21:36:53.846980 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0d5b74bea04f4f9545b3668f0faa2d" candidate_uuid: "9ff87a116bbd424c896d96c50e4e008d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1bb9b74bbd4acca5da79884c7d9273"
I20260926 21:36:53.847075 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0d5b74bea04f4f9545b3668f0faa2d" candidate_uuid: "9ff87a116bbd424c896d96c50e4e008d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d6d31ed2fa4e8793b81619badaf61c"
I20260926 21:36:53.847191 2425 raft_consensus.cc:3037] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:53.847258 2569 raft_consensus.cc:3037] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:53.850203 2425 raft_consensus.cc:2454] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff87a116bbd424c896d96c50e4e008d in term 1.
I20260926 21:36:53.850203 2569 raft_consensus.cc:2454] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff87a116bbd424c896d96c50e4e008d in term 1.
I20260926 21:36:53.850585 2219 leader_election.cc:302] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1bb9b74bbd4acca5da79884c7d9273, 9ff87a116bbd424c896d96c50e4e008d; no voters:
I20260926 21:36:53.850888 2734 raft_consensus.cc:2781] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:53.851068 2734 raft_consensus.cc:686] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 1 LEADER]: Becoming Leader. State: Replica: 9ff87a116bbd424c896d96c50e4e008d, State: Running, Role: LEADER
I20260926 21:36:53.851336 2734 consensus_queue.cc:237] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } }
I20260926 21:36:53.854295 2128 catalog_manager.cc:5495] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9ff87a116bbd424c896d96c50e4e008d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff87a116bbd424c896d96c50e4e008d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260926 21:36:53.931358 2569 raft_consensus.cc:1261] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Refusing update from remote peer 9ff87a116bbd424c896d96c50e4e008d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:53.931484 2425 raft_consensus.cc:1261] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Refusing update from remote peer 9ff87a116bbd424c896d96c50e4e008d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:36:53.931998 2734 consensus_queue.cc:1035] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1bb9b74bbd4acca5da79884c7d9273" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:36:53.932276 2743 consensus_queue.cc:1035] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d6d31ed2fa4e8793b81619badaf61c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 21:36:53.954694 2695 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 21:36:55.544776 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:55.545034 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:36:55.549391 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17056a7b8ff642449b31b9374be663a7: Sending DeleteTablet for 3 replicas of tablet 3a0d5b74bea04f4f9545b3668f0faa2d
I20260926 21:36:55.550375 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0d5b74bea04f4f9545b3668f0faa2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46200
I20260926 21:36:55.550559 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0d5b74bea04f4f9545b3668f0faa2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37358
I20260926 21:36:55.550565 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0d5b74bea04f4f9545b3668f0faa2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:36:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33394
I20260926 21:36:55.550938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2200
I20260926 21:36:55.562134 2775 tablet_replica.cc:330] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: stopping tablet replica
I20260926 21:36:55.562688 2777 tablet_replica.cc:330] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: stopping tablet replica
I20260926 21:36:55.563127 2777 raft_consensus.cc:2229] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:55.563488 2775 raft_consensus.cc:2229] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:36:55.563851 2777 raft_consensus.cc:2258] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:55.564152 2775 raft_consensus.cc:2258] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:55.566130 2775 ts_tablet_manager.cc:1905] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:55.566960 2777 ts_tablet_manager.cc:1905] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:36:55.570741 2776 tablet_replica.cc:330] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: stopping tablet replica
I20260926 21:36:55.573407 2776 raft_consensus.cc:2229] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:36:55.573897 2776 raft_consensus.cc:2258] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:36:55.577828 2777 ts_tablet_manager.cc:1918] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:36:55.577975 2777 log.cc:1196] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/3a0d5b74bea04f4f9545b3668f0faa2d
I20260926 21:36:55.578433 2777 ts_tablet_manager.cc:1939] T 3a0d5b74bea04f4f9545b3668f0faa2d P 0e1bb9b74bbd4acca5da79884c7d9273: Deleting consensus metadata
I20260926 21:36:55.579656 2116 catalog_manager.cc:4841] TS 0e1bb9b74bbd4acca5da79884c7d9273 (127.0.97.2:36219): tablet 3a0d5b74bea04f4f9545b3668f0faa2d (table test-table [id=8ec2ee7cd8fc4d7ebb77f839091135b4]) successfully deleted
I20260926 21:36:55.581839 2776 ts_tablet_manager.cc:1905] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 21:36:55 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458613, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:36:55.588263 2776 ts_tablet_manager.cc:1918] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:36:55.588551 2776 log.cc:1196] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/3a0d5b74bea04f4f9545b3668f0faa2d
I20260926 21:36:55.589164 2776 ts_tablet_manager.cc:1939] T 3a0d5b74bea04f4f9545b3668f0faa2d P 9ff87a116bbd424c896d96c50e4e008d: Deleting consensus metadata
I20260926 21:36:55.589159 2775 ts_tablet_manager.cc:1918] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 21:36:55.589318 2775 log.cc:1196] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/3a0d5b74bea04f4f9545b3668f0faa2d
I20260926 21:36:55.589831 2775 ts_tablet_manager.cc:1939] T 3a0d5b74bea04f4f9545b3668f0faa2d P b6d6d31ed2fa4e8793b81619badaf61c: Deleting consensus metadata
I20260926 21:36:55.590370 2116 catalog_manager.cc:4841] TS 9ff87a116bbd424c896d96c50e4e008d (127.0.97.1:36187): tablet 3a0d5b74bea04f4f9545b3668f0faa2d (table test-table [id=8ec2ee7cd8fc4d7ebb77f839091135b4]) successfully deleted
I20260926 21:36:55.590757 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:55.553401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.0.97.1:36187)
0926 21:36:55.563065 (+ 9664us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:55.563075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:55.567155 (+ 4080us) server_negotiation.cc:407] Connection header received
0926 21:36:55.567360 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:55.567369 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:55.567483 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:55.567669 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:55.569424 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:55.570603 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:55.572459 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:55.573207 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:55.573307 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:55.583171 (+ 9864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:55.583190 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:55.583215 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:55.583263 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:55.587249 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:55.588103 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:55.588113 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:55.588120 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:55.588288 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:55.588783 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:55.588789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:55.588795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:55.589327 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:55.589609 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:55.590167 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 21:36:55.590412 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9472,"thread_start_us":9279,"threads_started":1}
I20260926 21:36:55.591221 2114 catalog_manager.cc:4841] TS b6d6d31ed2fa4e8793b81619badaf61c (127.0.97.3:44527): tablet 3a0d5b74bea04f4f9545b3668f0faa2d (table test-table [id=8ec2ee7cd8fc4d7ebb77f839091135b4]) successfully deleted
W20260926 21:36:55.592195 2331 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37384
W20260926 21:36:55.592907 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2200: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:36:55.593142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 2200
I20260926 21:36:55.616089 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2340
Sep 26 21:36:55 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458613, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:36:55.631163 2780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:55.616862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.0.97.2:36219)
0926 21:36:55.617414 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:55.617419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:55.617433 (+ 14us) server_negotiation.cc:407] Connection header received
0926 21:36:55.617482 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:55.617485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:55.617562 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:55.617680 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:55.618886 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:55.619643 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:55.620884 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:55.621351 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:55.621476 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:55.625580 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:55.625595 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:55.625611 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:55.625649 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:55.628448 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:55.629122 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:55.629129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:55.629132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:55.629315 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:55.629675 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:55.629681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:55.629684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:55.630024 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:55.630311 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:55.630605 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 21:36:55.630819 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":278,"threads_started":1}
W20260926 21:36:55.631853 2470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33426
W20260926 21:36:55.632459 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2340: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:36:55.632606 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 2340
I20260926 21:36:55.654608 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2479
Sep 26 21:36:55 dist-test-slave-lhhq krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458613, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:36:55.668812 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:55.655338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.0.97.3:44527)
0926 21:36:55.655831 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:55.655835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:55.655851 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:36:55.655902 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:55.655905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:55.655961 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:55.656055 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:55.657280 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:55.657980 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:55.659148 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:55.659579 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:55.659671 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:55.663615 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:55.663629 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:55.663645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:55.663677 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:55.666469 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:55.667048 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:55.667055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:55.667058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:55.667178 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:55.667471 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:55.667474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:55.667476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:55.667817 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:55.668043 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:55.668224 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 21:36:55.668381 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":220,"threads_started":1}
W20260926 21:36:55.669727 2609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46212
W20260926 21:36:55.670367 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2479: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:36:55.670540 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 2479
I20260926 21:36:55.691556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2097
I20260926 21:36:55.692737 2169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:55.811878 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 2097
2026-09-26T21:36:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8613 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:36:55 dist-test-slave-lhhq krb5kdc[2785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:36:55 dist-test-slave-lhhq krb5kdc[2785](info): set up 2 sockets
Sep 26 21:36:55 dist-test-slave-lhhq krb5kdc[2785](info): commencing operation
krb5kdc: starting...
W20260926 21:36:56.897454 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:36:57 dist-test-slave-lhhq krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:36:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:36:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:36:57.084867 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33709
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:57.403079 2798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:57.403368 2798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:57.403551 2798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:57.410753 2798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:36:57.410889 2798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:57.410974 2798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:57.411075 2798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:36:57.411159 2798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:36:57.420799 2798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:57.422171 2798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:57.423458 2798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:57.435118 2803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:57.435106 2804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:57.435269 2806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:57.436120 2798 server_base.cc:1034] running on GCE node
I20260926 21:36:57.436857 2798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:57.438364 2798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:57.439565 2798 hybrid_clock.cc:648] HybridClock initialized: now 1790458617439508 us; error 58 us; skew 500 ppm
I20260926 21:36:57.439872 2798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:57 dist-test-slave-lhhq krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:57.442996 2798 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:36:57.444417 2798 webserver.cc:460] Webserver started at http://127.0.97.62:35007/ using document root <none> and password file <none>
I20260926 21:36:57.445065 2798 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:57.445160 2798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:57.445533 2798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:57.448446 2798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "9dbee63a2ba542528afe31e901f47eac"
format_stamp: "Formatted at 2026-09-26 21:36:57 on dist-test-slave-lhhq"
server_key: "9fd67fc8b189c0637c2e49c1b0eae631"
server_key_iv: "75354195eb3f7f9350df130d1ff8ceb9"
server_key_version: "encryptionkey@0"
I20260926 21:36:57.449167 2798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "9dbee63a2ba542528afe31e901f47eac"
format_stamp: "Formatted at 2026-09-26 21:36:57 on dist-test-slave-lhhq"
server_key: "9fd67fc8b189c0637c2e49c1b0eae631"
server_key_iv: "75354195eb3f7f9350df130d1ff8ceb9"
server_key_version: "encryptionkey@0"
I20260926 21:36:57.455431 2798 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 21:36:57.461549 2813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:57.463496 2798 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 21:36:57.463713 2798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "9dbee63a2ba542528afe31e901f47eac"
format_stamp: "Formatted at 2026-09-26 21:36:57 on dist-test-slave-lhhq"
server_key: "9fd67fc8b189c0637c2e49c1b0eae631"
server_key_iv: "75354195eb3f7f9350df130d1ff8ceb9"
server_key_version: "encryptionkey@0"
I20260926 21:36:57.463941 2798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:57.486234 2798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:57.492180 2798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:57.492515 2798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:57.514379 2798 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33709
I20260926 21:36:57.514413 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33709 every 8 connection(s)
I20260926 21:36:57.515775 2798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:36:57.519683 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:36:57.523195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 2798
I20260926 21:36:57.523424 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:36:57.524266 388 external_mini_cluster.cc:1428] Setting key b5fc55e29ba3ea49560463eb9ac0cc1b
I20260926 21:36:57.532981 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac: Bootstrap starting.
I20260926 21:36:57.536098 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac: Neither blocks nor log segments found. Creating new log.
I20260926 21:36:57.537118 2865 log.cc:824] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:36:57 dist-test-slave-lhhq krb5kdc[2785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:57.540115 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac: No bootstrap required, opened a new log
I20260926 21:36:57.546979 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:57.526727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.0.97.62:33709)
0926 21:36:57.527492 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:57.527505 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:57.527550 (+ 45us) server_negotiation.cc:407] Connection header received
0926 21:36:57.529343 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:57.529369 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:57.529824 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:57.530278 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:57.531811 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:57.533237 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:57.534517 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:57.535220 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:57.535419 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:57.539515 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:57.539523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:57.539545 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:57.539581 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:57.542750 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:57.543476 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:57.543483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:57.543490 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:57.543665 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:57.544050 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:57.544057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:57.544060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:57.544641 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:57.544975 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:57.545457 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 21:36:57.545901 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":335,"threads_started":1}
I20260926 21:36:57.548473 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbee63a2ba542528afe31e901f47eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33709 } }
I20260926 21:36:57.548897 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:36:57.549019 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbee63a2ba542528afe31e901f47eac, State: Initialized, Role: FOLLOWER
I20260926 21:36:57.549652 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbee63a2ba542528afe31e901f47eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33709 } }
I20260926 21:36:57.549921 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:36:57.550073 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:36:57.550272 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:36:57.552469 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbee63a2ba542528afe31e901f47eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33709 } }
I20260926 21:36:57.552999 2865 leader_election.cc:302] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dbee63a2ba542528afe31e901f47eac; no voters:
I20260926 21:36:57.553642 2865 leader_election.cc:288] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:36:57.553783 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:36:57.554503 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [term 1 LEADER]: Becoming Leader. State: Replica: 9dbee63a2ba542528afe31e901f47eac, State: Running, Role: LEADER
I20260926 21:36:57.555052 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbee63a2ba542528afe31e901f47eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33709 } }
I20260926 21:36:57.555703 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:36:57.558188 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dbee63a2ba542528afe31e901f47eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbee63a2ba542528afe31e901f47eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33709 } } }
I20260926 21:36:57.558593 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:57.559072 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbee63a2ba542528afe31e901f47eac. Latest consensus state: current_term: 1 leader_uuid: "9dbee63a2ba542528afe31e901f47eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbee63a2ba542528afe31e901f47eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33709 } } }
I20260926 21:36:57.559378 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac [sys.catalog]: This master's current role is: LEADER
I20260926 21:36:57.559695 2877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:36:57.567636 2877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:36:57.577270 2877 catalog_manager.cc:1347] Generated new cluster ID: 7160a412484d41dcad32850bc47e2b07
I20260926 21:36:57.577397 2877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:36:57.599099 2877 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:36:57.600566 2877 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:36:57.617724 2877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dbee63a2ba542528afe31e901f47eac: Generated new TSK 0
I20260926 21:36:57.618872 2877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:36:57.666957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33709
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:57.993117 2892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:57.993410 2892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:57.993502 2892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:58.000699 2892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:58.000821 2892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:58.000932 2892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:36:58.010460 2892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:58.011703 2892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:58.012998 2892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:58.025823 2897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:58.025842 2900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:58.025789 2898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:58.026903 2892 server_base.cc:1034] running on GCE node
I20260926 21:36:58.027505 2892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:58.028434 2892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:58.029654 2892 hybrid_clock.cc:648] HybridClock initialized: now 1790458618029601 us; error 46 us; skew 500 ppm
I20260926 21:36:58.029979 2892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:58 dist-test-slave-lhhq krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:58.032967 2892 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:36:58.034422 2892 webserver.cc:460] Webserver started at http://127.0.97.1:39241/ using document root <none> and password file <none>
I20260926 21:36:58.035094 2892 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:58.035238 2892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:58.035544 2892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:58.038426 2892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "9bb9b2e6bbdd4b1fb38856053140ef07"
format_stamp: "Formatted at 2026-09-26 21:36:58 on dist-test-slave-lhhq"
server_key: "5d71f72c0aba76288db8c4d7b6ff61a9"
server_key_iv: "404338e9bbb5c80da6293fc1f481055a"
server_key_version: "encryptionkey@0"
I20260926 21:36:58.039219 2892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9bb9b2e6bbdd4b1fb38856053140ef07"
format_stamp: "Formatted at 2026-09-26 21:36:58 on dist-test-slave-lhhq"
server_key: "5d71f72c0aba76288db8c4d7b6ff61a9"
server_key_iv: "404338e9bbb5c80da6293fc1f481055a"
server_key_version: "encryptionkey@0"
I20260926 21:36:58.044299 2892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 21:36:58.047878 2907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:58.049155 2892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:36:58.049384 2892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "9bb9b2e6bbdd4b1fb38856053140ef07"
format_stamp: "Formatted at 2026-09-26 21:36:58 on dist-test-slave-lhhq"
server_key: "5d71f72c0aba76288db8c4d7b6ff61a9"
server_key_iv: "404338e9bbb5c80da6293fc1f481055a"
server_key_version: "encryptionkey@0"
I20260926 21:36:58.049597 2892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:58.064524 2892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:58.069377 2892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:58.069675 2892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:58.070637 2892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:36:58.072001 2892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:58.072116 2892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:58.072263 2892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:58.072338 2892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:58.105582 2892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39119
I20260926 21:36:58.105628 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39119 every 8 connection(s)
I20260926 21:36:58.106930 2892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:36:58.115392 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 2892
I20260926 21:36:58.115613 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:36:58.116223 388 external_mini_cluster.cc:1428] Setting key 775bdd0620905c02a792eefd9cd54b83
Sep 26 21:36:58 dist-test-slave-lhhq krb5kdc[2785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:58.128836 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:58.109561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57123 (local address 127.0.97.62:33709)
0926 21:36:58.110003 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:58.110007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:58.110934 (+ 927us) server_negotiation.cc:407] Connection header received
0926 21:36:58.112719 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:58.112725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:58.112813 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:58.112929 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:58.114457 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:58.115227 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:58.116998 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:58.117660 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:58.117782 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:58.121729 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:58.121738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:58.121746 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:58.121790 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:58.125806 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:58.126560 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:58.126565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:58.126569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:58.126696 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:58.127123 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:58.127129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:58.127132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:58.127407 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:58.127568 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:58.128249 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 21:36:58.128464 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":339,"thread_start_us":242,"threads_started":1}
I20260926 21:36:58.129999 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:58.110062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33709 (local address 127.0.97.1:57123)
0926 21:36:58.110771 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:58.110814 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:58.112452 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:58.113165 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:58.113173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:58.113728 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:58.114292 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:58.114308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:58.115388 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:58.115392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:58.116823 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:58.116838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:58.117930 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:58.117940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:58.118243 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:58.118995 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:58.119018 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:58.121475 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:58.125956 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:58.125964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:58.125979 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:58.126424 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:58.126805 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:58.126811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:58.126814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:58.127003 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:58.127534 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:58.127543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:58.127893 (+ 350us) client_negotiation.cc:769] Sending connection context
0926 21:36:58.128161 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 21:36:58.128695 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":283,"threads_started":1}
I20260926 21:36:58.131608 3020 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33709
I20260926 21:36:58.131924 3020 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:58.132706 3020 heartbeater.cc:511] Master 127.0.97.62:33709 requested a full tablet report, sending...
I20260926 21:36:58.135138 2830 ts_manager.cc:194] Registered new tserver with Master: 9bb9b2e6bbdd4b1fb38856053140ef07 (127.0.97.1:39119)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:36:58.137099 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:36:58.162140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33709
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:58.484055 3026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:58.484416 3026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:58.484581 3026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:58.491950 3026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:58.492106 3026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:58.492282 3026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:36:58.502149 3026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:58.503506 3026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:58.504830 3026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:58.517824 3032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:58.517935 3034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:58.518448 3031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:58.519148 3026 server_base.cc:1034] running on GCE node
I20260926 21:36:58.519757 3026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:58.520691 3026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:58.521908 3026 hybrid_clock.cc:648] HybridClock initialized: now 1790458618521855 us; error 46 us; skew 500 ppm
I20260926 21:36:58.522225 3026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:58 dist-test-slave-lhhq krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:58.525497 3026 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:36:58.526983 3026 webserver.cc:460] Webserver started at http://127.0.97.2:33799/ using document root <none> and password file <none>
I20260926 21:36:58.527685 3026 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:58.527827 3026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:58.528131 3026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:58.531060 3026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "f7d21b178a7145c386e97489718d6cdb"
format_stamp: "Formatted at 2026-09-26 21:36:58 on dist-test-slave-lhhq"
server_key: "4ecd1a91e1f85b704309a4d1466092ff"
server_key_iv: "34b6c6b4d1fc947e65cabad32aa74091"
server_key_version: "encryptionkey@0"
I20260926 21:36:58.531844 3026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f7d21b178a7145c386e97489718d6cdb"
format_stamp: "Formatted at 2026-09-26 21:36:58 on dist-test-slave-lhhq"
server_key: "4ecd1a91e1f85b704309a4d1466092ff"
server_key_iv: "34b6c6b4d1fc947e65cabad32aa74091"
server_key_version: "encryptionkey@0"
I20260926 21:36:58.536940 3026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:36:58.540450 3041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:58.541853 3026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:36:58.542030 3026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "f7d21b178a7145c386e97489718d6cdb"
format_stamp: "Formatted at 2026-09-26 21:36:58 on dist-test-slave-lhhq"
server_key: "4ecd1a91e1f85b704309a4d1466092ff"
server_key_iv: "34b6c6b4d1fc947e65cabad32aa74091"
server_key_version: "encryptionkey@0"
I20260926 21:36:58.542243 3026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:58.554158 3026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:58.558136 3026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:58.558414 3026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:58.559365 3026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:36:58.560640 3026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:58.560740 3026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:58.560889 3026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:58.560961 3026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:58.593580 3026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43429
I20260926 21:36:58.593627 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43429 every 8 connection(s)
I20260926 21:36:58.594844 3026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:36:58.600173 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 3026
I20260926 21:36:58.600354 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:36:58.601033 388 external_mini_cluster.cc:1428] Setting key 64e730bbcbd2715a69238efb6c4ab8d5
Sep 26 21:36:58 dist-test-slave-lhhq krb5kdc[2785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:58.614449 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:58.597299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47133 (local address 127.0.97.62:33709)
0926 21:36:58.597459 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:58.597463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:58.598567 (+ 1104us) server_negotiation.cc:407] Connection header received
0926 21:36:58.600338 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:58.600343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:58.600451 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:58.600580 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:58.602205 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:58.603015 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:58.604573 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:58.605347 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:58.605492 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:58.608915 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:58.608921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:58.608924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:58.608954 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:58.611494 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:58.612224 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:58.612237 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:58.612239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:58.612359 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:58.612785 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:58.612790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:58.612794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:58.613042 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:58.613220 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:58.613892 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 21:36:58.614114 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 21:36:58.615458 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:58.597738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33709 (local address 127.0.97.2:47133)
0926 21:36:58.598405 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:58.598447 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:58.600085 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:58.600804 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:58.600812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:58.601378 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:58.602043 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:58.602062 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:58.603180 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:58.603184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:58.604406 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:58.604422 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:58.605558 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:58.605563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:58.605759 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:58.606446 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:58.606468 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:58.608688 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:58.611648 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:58.611657 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:58.611676 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:58.612099 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:58.612466 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:58.612472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:58.612475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:58.612666 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:58.613165 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:58.613173 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:58.613533 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 21:36:58.613814 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 21:36:58.614354 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":277,"threads_started":1}
I20260926 21:36:58.616974 3154 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33709
I20260926 21:36:58.617329 3154 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:58.618019 3154 heartbeater.cc:511] Master 127.0.97.62:33709 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:36:58.619463 2830 ts_manager.cc:194] Registered new tserver with Master: f7d21b178a7145c386e97489718d6cdb (127.0.97.2:43429)
I20260926 21:36:58.620236 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:36:58.644040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33709
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:36:58.960052 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:36:58.960318 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:36:58.960409 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:36:58.967621 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:36:58.967759 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:36:58.967873 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:36:58.977600 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:36:58.978832 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:36:58.980152 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:36:58.993362 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:58.993286 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:36:58.993330 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:36:58.994187 3159 server_base.cc:1034] running on GCE node
I20260926 21:36:58.994736 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:36:58.995689 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:36:58.996881 3159 hybrid_clock.cc:648] HybridClock initialized: now 1790458618996821 us; error 65 us; skew 500 ppm
I20260926 21:36:58.997227 3159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:36:58 dist-test-slave-lhhq krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:36:59.000314 3159 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:36:59.001900 3159 webserver.cc:460] Webserver started at http://127.0.97.3:35643/ using document root <none> and password file <none>
I20260926 21:36:59.002583 3159 fs_manager.cc:359] Metadata directory not provided
I20260926 21:36:59.002712 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:36:59.003028 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:36:59.005908 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "bac6ff61d00649e884cb3428d2ddebc1"
format_stamp: "Formatted at 2026-09-26 21:36:58 on dist-test-slave-lhhq"
server_key: "26acccc1abfa10003003ee0dad9bd2b5"
server_key_iv: "d068ece01d98643e0095e9979b286da1"
server_key_version: "encryptionkey@0"
I20260926 21:36:59.006667 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bac6ff61d00649e884cb3428d2ddebc1"
format_stamp: "Formatted at 2026-09-26 21:36:58 on dist-test-slave-lhhq"
server_key: "26acccc1abfa10003003ee0dad9bd2b5"
server_key_iv: "d068ece01d98643e0095e9979b286da1"
server_key_version: "encryptionkey@0"
I20260926 21:36:59.011597 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:36:59.014859 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:59.016134 3159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:36:59.016300 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "bac6ff61d00649e884cb3428d2ddebc1"
format_stamp: "Formatted at 2026-09-26 21:36:58 on dist-test-slave-lhhq"
server_key: "26acccc1abfa10003003ee0dad9bd2b5"
server_key_iv: "d068ece01d98643e0095e9979b286da1"
server_key_version: "encryptionkey@0"
I20260926 21:36:59.016494 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:36:59.051260 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:36:59.055482 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:36:59.055724 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:36:59.056583 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:36:59.057912 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:36:59.057986 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:59.058077 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:36:59.058113 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:36:59.090179 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34401
I20260926 21:36:59.090226 3286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34401 every 8 connection(s)
I20260926 21:36:59.091449 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:36:59.092442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 3159
I20260926 21:36:59.092630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:36:59.093134 388 external_mini_cluster.cc:1428] Setting key 0c86e6eb81d03a2a1a29c42787b1f89f
Sep 26 21:36:59 dist-test-slave-lhhq krb5kdc[2785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:36:59.112066 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:59.094290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48375 (local address 127.0.97.62:33709)
0926 21:36:59.094466 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:59.094472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:59.096030 (+ 1558us) server_negotiation.cc:407] Connection header received
0926 21:36:59.097894 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:59.097898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:59.097961 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:59.098041 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:59.099437 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:59.100145 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:59.101677 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:36:59.102037 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:36:59.102137 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:59.105554 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:36:59.105561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:36:59.105565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:36:59.105607 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:36:59.109291 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:59.110001 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:59.110006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:59.110009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:59.110155 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:36:59.110563 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:36:59.110569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:36:59.110572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:36:59.110928 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:36:59.111069 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 21:36:59.111680 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 21:36:59.111835 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:36:59.113015 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:36:59.095001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33709 (local address 127.0.97.3:48375)
0926 21:36:59.095870 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 21:36:59.095919 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 21:36:59.097659 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:36:59.098243 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:36:59.098250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:36:59.098744 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:36:59.099299 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:59.099315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:59.100270 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:59.100274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:59.101549 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:36:59.101560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:36:59.102230 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:36:59.102235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:36:59.102419 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:36:59.103120 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:36:59.103145 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:36:59.105338 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:36:59.109423 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:59.109429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:59.109443 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:59.109866 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:59.110266 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:36:59.110269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:36:59.110271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:36:59.110442 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:36:59.111042 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:36:59.111048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:36:59.111351 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 21:36:59.111567 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 21:36:59.111929 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":371,"threads_started":1}
I20260926 21:36:59.114436 3287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33709
I20260926 21:36:59.114701 3287 heartbeater.cc:464] Registering TS with master...
I20260926 21:36:59.115398 3287 heartbeater.cc:511] Master 127.0.97.62:33709 requested a full tablet report, sending...
I20260926 21:36:59.116744 2830 ts_manager.cc:194] Registered new tserver with Master: bac6ff61d00649e884cb3428d2ddebc1 (127.0.97.3:34401)
I20260926 21:36:59.117586 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48375
I20260926 21:36:59.120831 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:36:59.140600 3020 heartbeater.cc:503] Master 127.0.97.62:33709 was elected leader, sending a full tablet report...
W20260926 21:36:59.150578 3022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:36:59.148375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.0.97.62:33709)
0926 21:36:59.148549 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:36:59.148582 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 21:36:59.148986 (+ 404us) server_negotiation.cc:407] Connection header received
0926 21:36:59.149131 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:36:59.149135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:36:59.149236 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:36:59.149338 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:36:59.150384 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33482: BlockingRecv error: recv got EOF from 127.0.0.1:33482 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 21:36:59.154066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2892
W20260926 21:36:59.156785 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2892: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39119: server requires authentication, but client does not have Kerberos credentials available
I20260926 21:36:59.156977 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 2892
I20260926 21:36:59.172590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3026
W20260926 21:36:59.174998 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3026: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:43429: server requires authentication, but client does not have Kerberos credentials available
I20260926 21:36:59.175113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 3026
I20260926 21:36:59.190045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3159
W20260926 21:36:59.192595 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3159: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34401: server requires authentication, but client does not have Kerberos credentials available
I20260926 21:36:59.192744 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 3159
I20260926 21:36:59.208132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 2798
I20260926 21:36:59.209398 2860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:36:59.310026 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 2798
2026-09-26T21:36:59Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3495 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:36:59 dist-test-slave-lhhq krb5kdc[3304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:36:59 dist-test-slave-lhhq krb5kdc[3304](info): set up 2 sockets
Sep 26 21:36:59 dist-test-slave-lhhq krb5kdc[3304](info): commencing operation
krb5kdc: starting...
W20260926 21:37:00.405637 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:37:00 dist-test-slave-lhhq krb5kdc[3304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:37:00.598623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43557
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:00.915387 3317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:00.915668 3317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:00.915827 3317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:00.922978 3317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:00.923094 3317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:00.923177 3317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:00.923267 3317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:00.923353 3317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:00.933051 3317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:00.934351 3317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:00.935561 3317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:00.946767 3323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:00.947082 3322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:00.947045 3325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:00.947681 3317 server_base.cc:1034] running on GCE node
I20260926 21:37:00.948469 3317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:00.949893 3317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:00.951151 3317 hybrid_clock.cc:648] HybridClock initialized: now 1790458620951097 us; error 44 us; skew 500 ppm
I20260926 21:37:00.951519 3317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:00 dist-test-slave-lhhq krb5kdc[3304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:00.959326 3317 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:37:00.960826 3317 webserver.cc:460] Webserver started at http://127.0.97.62:40627/ using document root <none> and password file <none>
I20260926 21:37:00.961573 3317 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:00.961715 3317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:00.962015 3317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:00.964874 3317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "1c5e4899a8324bb485d806638dd4f166"
format_stamp: "Formatted at 2026-09-26 21:37:00 on dist-test-slave-lhhq"
server_key: "a81463580693d520313b5376ba3594d1"
server_key_iv: "cadcd9a2270a59c3ad161d59b5f05d08"
server_key_version: "encryptionkey@0"
I20260926 21:37:00.965677 3317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c5e4899a8324bb485d806638dd4f166"
format_stamp: "Formatted at 2026-09-26 21:37:00 on dist-test-slave-lhhq"
server_key: "a81463580693d520313b5376ba3594d1"
server_key_iv: "cadcd9a2270a59c3ad161d59b5f05d08"
server_key_version: "encryptionkey@0"
I20260926 21:37:00.970700 3317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 21:37:00.973999 3332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:00.975286 3317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 21:37:00.975476 3317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "1c5e4899a8324bb485d806638dd4f166"
format_stamp: "Formatted at 2026-09-26 21:37:00 on dist-test-slave-lhhq"
server_key: "a81463580693d520313b5376ba3594d1"
server_key_iv: "cadcd9a2270a59c3ad161d59b5f05d08"
server_key_version: "encryptionkey@0"
I20260926 21:37:00.975680 3317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:01.006523 3317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:01.010533 3317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:01.010823 3317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:01.028754 3317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43557
I20260926 21:37:01.028782 3383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43557 every 8 connection(s)
I20260926 21:37:01.030293 3317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:01.033941 3384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:01.036835 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 3317
I20260926 21:37:01.037060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:01.037871 388 external_mini_cluster.cc:1428] Setting key 823e49722cb9ff0a1b11795c901fbefb
I20260926 21:37:01.046285 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166: Bootstrap starting.
I20260926 21:37:01.050056 3384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:01.051390 3384 log.cc:824] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:37:01 dist-test-slave-lhhq krb5kdc[3304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:01.054626 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166: No bootstrap required, opened a new log
I20260926 21:37:01.062420 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:01.039776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.0.97.62:43557)
0926 21:37:01.040553 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:01.040563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:01.040603 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:37:01.042268 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:01.042290 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:01.042699 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:01.043029 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:01.044844 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:01.046235 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:01.048006 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:01.048701 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:01.048831 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:01.053718 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:01.053730 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:01.053748 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:01.053796 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:01.057738 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:01.058429 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:01.058437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:01.058445 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:01.058659 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:01.059027 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:01.059037 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:01.059040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:01.059760 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:01.060062 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:01.060474 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 21:37:01.060940 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":374,"threads_started":1}
I20260926 21:37:01.064682 3384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e4899a8324bb485d806638dd4f166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43557 } }
I20260926 21:37:01.065080 3384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:01.065214 3384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5e4899a8324bb485d806638dd4f166, State: Initialized, Role: FOLLOWER
I20260926 21:37:01.065850 3384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e4899a8324bb485d806638dd4f166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43557 } }
I20260926 21:37:01.066104 3384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:01.066250 3384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:01.066431 3384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:01.066463 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:37:01.069582 3384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e4899a8324bb485d806638dd4f166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43557 } }
I20260926 21:37:01.070179 3384 leader_election.cc:302] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c5e4899a8324bb485d806638dd4f166; no voters:
I20260926 21:37:01.070864 3384 leader_election.cc:288] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:01.071030 3389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:01.073271 3384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:01.077589 3389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [term 1 LEADER]: Becoming Leader. State: Replica: 1c5e4899a8324bb485d806638dd4f166, State: Running, Role: LEADER
I20260926 21:37:01.078245 3389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e4899a8324bb485d806638dd4f166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43557 } }
I20260926 21:37:01.081209 3390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c5e4899a8324bb485d806638dd4f166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e4899a8324bb485d806638dd4f166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43557 } } }
I20260926 21:37:01.081584 3390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:01.082069 3391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c5e4899a8324bb485d806638dd4f166. Latest consensus state: current_term: 1 leader_uuid: "1c5e4899a8324bb485d806638dd4f166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e4899a8324bb485d806638dd4f166" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43557 } } }
I20260926 21:37:01.082351 3391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:01.086776 3401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:01.091744 3401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:01.103609 3401 catalog_manager.cc:1347] Generated new cluster ID: 2d696e6fc8f04f36a9598e7e01fee1bd
I20260926 21:37:01.103720 3401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:01.119939 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:01.111019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.0.97.62:43557)
0926 21:37:01.111192 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:01.111199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:01.111778 (+ 579us) server_negotiation.cc:407] Connection header received
0926 21:37:01.111965 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:01.111970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:01.112053 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:01.112157 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:01.113464 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:01.114559 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:01.116368 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:01.116982 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:01.117090 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:01.118299 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:01.118307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:01.118312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:01.119155 (+ 843us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:01.119222 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:01.119257 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:01.119354 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:01.119514 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 21:37:01.119606 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:37:01.126389 3401 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:01.128072 3401 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:01.151645 3401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c5e4899a8324bb485d806638dd4f166: Generated new TSK 0
I20260926 21:37:01.152741 3401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:37:01.162195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3317
I20260926 21:37:01.163287 3379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:01.262029 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 3317
2026-09-26T21:37:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1946 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:37:01 dist-test-slave-lhhq krb5kdc[3418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:37:01 dist-test-slave-lhhq krb5kdc[3418](info): set up 2 sockets
Sep 26 21:37:01 dist-test-slave-lhhq krb5kdc[3418](info): commencing operation
krb5kdc: starting...
W20260926 21:37:02.345578 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:37:02 dist-test-slave-lhhq krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:37:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:37:02.540587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45331
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:02.862784 3431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:02.863101 3431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:02.863190 3431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:02.870563 3431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:02.870679 3431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:02.870725 3431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:02.870782 3431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:02.870826 3431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:02.880648 3431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:02.881965 3431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:02.883209 3431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:02.894815 3439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:02.894881 3437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:02.894786 3436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:02.898810 3431 server_base.cc:1034] running on GCE node
I20260926 21:37:02.899680 3431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:02.901167 3431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:02.902443 3431 hybrid_clock.cc:648] HybridClock initialized: now 1790458622902399 us; error 41 us; skew 500 ppm
I20260926 21:37:02.902799 3431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:02 dist-test-slave-lhhq krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:02.905925 3431 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:37:02.907394 3431 webserver.cc:460] Webserver started at http://127.0.97.62:43103/ using document root <none> and password file <none>
I20260926 21:37:02.908061 3431 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:02.908156 3431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:02.908399 3431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:02.911316 3431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "97752b0cdf3c4176ad6300700dbba2ec"
format_stamp: "Formatted at 2026-09-26 21:37:02 on dist-test-slave-lhhq"
server_key: "31c971fcbe2053280d8f705ba99437d5"
server_key_iv: "c3e8f1f0cfac772d5f07c7f2803f241d"
server_key_version: "encryptionkey@0"
I20260926 21:37:02.912006 3431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "97752b0cdf3c4176ad6300700dbba2ec"
format_stamp: "Formatted at 2026-09-26 21:37:02 on dist-test-slave-lhhq"
server_key: "31c971fcbe2053280d8f705ba99437d5"
server_key_iv: "c3e8f1f0cfac772d5f07c7f2803f241d"
server_key_version: "encryptionkey@0"
I20260926 21:37:02.916999 3431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 21:37:02.920320 3446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:02.921852 3431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:02.922006 3431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "97752b0cdf3c4176ad6300700dbba2ec"
format_stamp: "Formatted at 2026-09-26 21:37:02 on dist-test-slave-lhhq"
server_key: "31c971fcbe2053280d8f705ba99437d5"
server_key_iv: "c3e8f1f0cfac772d5f07c7f2803f241d"
server_key_version: "encryptionkey@0"
I20260926 21:37:02.922185 3431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:02.944649 3431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:02.948805 3431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:02.949056 3431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:02.967813 3431 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45331
I20260926 21:37:02.967854 3497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45331 every 8 connection(s)
I20260926 21:37:02.969266 3431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:02.973007 3498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:02.979296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 3431
I20260926 21:37:02.979548 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:02.980350 388 external_mini_cluster.cc:1428] Setting key 1be35bd6940a790227a55a7183be1dff
I20260926 21:37:02.984877 3498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec: Bootstrap starting.
I20260926 21:37:02.988157 3498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:02.989305 3498 log.cc:824] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:02.992317 3498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec: No bootstrap required, opened a new log
Sep 26 21:37:02 dist-test-slave-lhhq krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:03.000835 3498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97752b0cdf3c4176ad6300700dbba2ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260926 21:37:03.001305 3498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:03.001436 3498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97752b0cdf3c4176ad6300700dbba2ec, State: Initialized, Role: FOLLOWER
I20260926 21:37:03.002193 3498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97752b0cdf3c4176ad6300700dbba2ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260926 21:37:03.002457 3498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:03.002624 3498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:03.002807 3498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:03.003720 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:02.982685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.0.97.62:45331)
0926 21:37:02.983470 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:02.983483 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:02.983525 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:37:02.985298 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:02.985321 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:02.985735 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:02.986158 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:02.987609 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:02.988971 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:02.990356 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:02.991026 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:02.991233 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:02.995567 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:02.995575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:02.995592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:02.995633 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:02.999058 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:02.999919 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:02.999927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:02.999935 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:03.000124 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:03.000519 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:03.000526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:03.000530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:03.001095 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:03.001531 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:03.001959 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 21:37:03.002463 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":344,"threads_started":1}
I20260926 21:37:03.005292 3498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97752b0cdf3c4176ad6300700dbba2ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260926 21:37:03.005808 3498 leader_election.cc:302] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97752b0cdf3c4176ad6300700dbba2ec; no voters:
I20260926 21:37:03.006497 3498 leader_election.cc:288] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:03.006603 3503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:03.007493 3503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [term 1 LEADER]: Becoming Leader. State: Replica: 97752b0cdf3c4176ad6300700dbba2ec, State: Running, Role: LEADER
I20260926 21:37:03.008051 3503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97752b0cdf3c4176ad6300700dbba2ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260926 21:37:03.008850 3498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:03.011080 3505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97752b0cdf3c4176ad6300700dbba2ec. Latest consensus state: current_term: 1 leader_uuid: "97752b0cdf3c4176ad6300700dbba2ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97752b0cdf3c4176ad6300700dbba2ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } } }
I20260926 21:37:03.011152 3504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97752b0cdf3c4176ad6300700dbba2ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97752b0cdf3c4176ad6300700dbba2ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } } }
I20260926 21:37:03.011571 3505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:03.011615 3504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:03.013320 3508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:03.018644 3508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:03.027372 3508 catalog_manager.cc:1347] Generated new cluster ID: f186df6b12554bfe9690093712984a30
I20260926 21:37:03.027534 3508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:03.046907 3508 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:03.048858 3508 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:03.060515 3508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97752b0cdf3c4176ad6300700dbba2ec: Generated new TSK 0
I20260926 21:37:03.061667 3508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:37:03.125471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:03.459991 3524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:03.460330 3524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:03.460502 3524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:03.467770 3524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:03.467936 3524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:03.468135 3524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:03.478406 3524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:03.479802 3524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:03.481142 3524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:03.494442 3532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:03.494546 3530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:03.494552 3529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:03.495196 3524 server_base.cc:1034] running on GCE node
I20260926 21:37:03.495786 3524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:03.496694 3524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:03.497887 3524 hybrid_clock.cc:648] HybridClock initialized: now 1790458623497833 us; error 44 us; skew 500 ppm
I20260926 21:37:03.498211 3524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:03 dist-test-slave-lhhq krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:03.501441 3524 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:37:03.502918 3524 webserver.cc:460] Webserver started at http://127.0.97.1:34279/ using document root <none> and password file <none>
I20260926 21:37:03.503636 3524 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:03.503775 3524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:03.504073 3524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:03.507151 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "8d7c24a190ca4988b1c286cde75a3cb1"
format_stamp: "Formatted at 2026-09-26 21:37:03 on dist-test-slave-lhhq"
server_key: "57424890fe34cf5271fcca3ec0e257ec"
server_key_iv: "eb7827c38f19d27a4902b288e7c819de"
server_key_version: "encryptionkey@0"
I20260926 21:37:03.507968 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8d7c24a190ca4988b1c286cde75a3cb1"
format_stamp: "Formatted at 2026-09-26 21:37:03 on dist-test-slave-lhhq"
server_key: "57424890fe34cf5271fcca3ec0e257ec"
server_key_iv: "eb7827c38f19d27a4902b288e7c819de"
server_key_version: "encryptionkey@0"
I20260926 21:37:03.513177 3524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 21:37:03.516649 3539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:03.518028 3524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:03.518203 3524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "8d7c24a190ca4988b1c286cde75a3cb1"
format_stamp: "Formatted at 2026-09-26 21:37:03 on dist-test-slave-lhhq"
server_key: "57424890fe34cf5271fcca3ec0e257ec"
server_key_iv: "eb7827c38f19d27a4902b288e7c819de"
server_key_version: "encryptionkey@0"
I20260926 21:37:03.518409 3524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:03.553013 3524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:03.557230 3524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:03.557585 3524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:03.558674 3524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:03.560065 3524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:03.560170 3524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:03.560321 3524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:03.560392 3524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:03.592945 3524 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33331
I20260926 21:37:03.592988 3651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33331 every 8 connection(s)
I20260926 21:37:03.594291 3524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:03.596254 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 3524
I20260926 21:37:03.596513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:37:03.597280 388 external_mini_cluster.cc:1428] Setting key 7d6862bad41ee5785bd6e014eac87dc6
Sep 26 21:37:03 dist-test-slave-lhhq krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:37:03.618310 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:03.597339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57335 (local address 127.0.97.62:45331)
0926 21:37:03.597815 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:03.597822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:03.599006 (+ 1184us) server_negotiation.cc:407] Connection header received
0926 21:37:03.602042 (+ 3036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:03.602046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:03.602109 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:03.602202 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:03.604464 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:03.605170 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:03.606964 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:03.607389 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:03.607539 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:03.611777 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:03.611787 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:03.611794 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:03.611843 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:03.614799 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:03.615900 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:03.615904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:03.615906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:03.616017 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:03.616540 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:03.616543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:03.616545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:03.616790 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:03.616894 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:03.617887 (+ 993us) server_negotiation.cc:299] Negotiation successful
0926 21:37:03.618051 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":223,"threads_started":1}
I20260926 21:37:03.619441 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:03.597941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45331 (local address 127.0.97.1:57335)
0926 21:37:03.598849 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:03.598890 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:03.601707 (+ 2817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:03.602519 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:03.602536 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:03.603360 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:03.604255 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:03.604285 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:03.605379 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:03.605388 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:03.606767 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:03.606780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:03.607638 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:03.607647 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:03.607899 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:03.608693 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:03.608725 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:03.611505 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:03.615028 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:03.615041 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:03.615061 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:03.615758 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:03.616140 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:03.616148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:03.616154 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:03.616419 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:03.616944 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:03.616955 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:03.617453 (+ 498us) client_negotiation.cc:769] Sending connection context
0926 21:37:03.617832 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 21:37:03.618266 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":422,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:37:03.621304 3652 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45331
I20260926 21:37:03.621639 3652 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:03.622493 3652 heartbeater.cc:511] Master 127.0.97.62:45331 requested a full tablet report, sending...
I20260926 21:37:03.624867 3463 ts_manager.cc:194] Registered new tserver with Master: 8d7c24a190ca4988b1c286cde75a3cb1 (127.0.97.1:33331)
I20260926 21:37:03.626631 3463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:37:03.644589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:03.968452 3658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:03.968786 3658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:03.968993 3658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:03.976475 3658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:03.976651 3658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:03.976837 3658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:03.986878 3658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:03.988267 3658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:03.989709 3658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:04.003196 3666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:04.003295 3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:04.003247 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:04.003796 3658 server_base.cc:1034] running on GCE node
I20260926 21:37:04.004438 3658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:04.005483 3658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:04.006714 3658 hybrid_clock.cc:648] HybridClock initialized: now 1790458624006663 us; error 46 us; skew 500 ppm
I20260926 21:37:04.007031 3658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:04.010457 3658 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:37:04.011991 3658 webserver.cc:460] Webserver started at http://127.0.97.2:33839/ using document root <none> and password file <none>
I20260926 21:37:04.012692 3658 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:04.012822 3658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:04.013134 3658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:04.016238 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "656dd82b5b0a4f5eae0209e50c94156b"
format_stamp: "Formatted at 2026-09-26 21:37:04 on dist-test-slave-lhhq"
server_key: "913722ea4d50028baaea0cbdfd01d2d3"
server_key_iv: "231dca157bcc44181e8919f793cda144"
server_key_version: "encryptionkey@0"
I20260926 21:37:04.017016 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "656dd82b5b0a4f5eae0209e50c94156b"
format_stamp: "Formatted at 2026-09-26 21:37:04 on dist-test-slave-lhhq"
server_key: "913722ea4d50028baaea0cbdfd01d2d3"
server_key_iv: "231dca157bcc44181e8919f793cda144"
server_key_version: "encryptionkey@0"
I20260926 21:37:04.022446 3658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 21:37:04.026078 3673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:04.027567 3658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:04.027750 3658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "656dd82b5b0a4f5eae0209e50c94156b"
format_stamp: "Formatted at 2026-09-26 21:37:04 on dist-test-slave-lhhq"
server_key: "913722ea4d50028baaea0cbdfd01d2d3"
server_key_iv: "231dca157bcc44181e8919f793cda144"
server_key_version: "encryptionkey@0"
I20260926 21:37:04.027963 3658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:04.049177 3658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:04.053555 3658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:04.053848 3658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:04.054812 3658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:04.056092 3658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:04.056162 3658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:04.056332 3658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:04.056383 3658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:04.089561 3658 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41425
I20260926 21:37:04.089596 3785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41425 every 8 connection(s)
I20260926 21:37:04.090919 3658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:04.093381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 3658
I20260926 21:37:04.093587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:37:04.094110 388 external_mini_cluster.cc:1428] Setting key bb1d08c0677a28a180c02697d72bf8f9
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:37:04.113302 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:04.093752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46729 (local address 127.0.97.62:45331)
0926 21:37:04.093914 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:04.093919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:04.095460 (+ 1541us) server_negotiation.cc:407] Connection header received
0926 21:37:04.097586 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:04.097595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:04.097690 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:04.097807 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:04.099286 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.100444 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.102654 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.103092 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.103199 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:04.107433 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:04.107438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:04.107442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:04.107476 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:04.110263 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.111025 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.111036 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.111040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.111160 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.111590 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.111599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.111601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.111846 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:04.112006 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:04.112689 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 21:37:04.112903 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:37:04.114449 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:04.094391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45331 (local address 127.0.97.2:46729)
0926 21:37:04.095280 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:04.095331 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:04.097278 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:04.098017 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:04.098026 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:04.098526 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:04.099096 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:04.099112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.100597 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.100601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:04.102488 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:04.102506 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.103330 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.103339 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:04.103681 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:04.104438 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:04.104476 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:04.107177 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:04.110410 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:04.110420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:04.110438 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:04.110884 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:04.111273 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:04.111279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:04.111282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:04.111472 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:04.111972 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:04.111981 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:04.112334 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 21:37:04.112602 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 21:37:04.113142 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":344,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 21:37:04.116154 3786 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45331
I20260926 21:37:04.116456 3786 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:04.117178 3786 heartbeater.cc:511] Master 127.0.97.62:45331 requested a full tablet report, sending...
I20260926 21:37:04.118741 3463 ts_manager.cc:194] Registered new tserver with Master: 656dd82b5b0a4f5eae0209e50c94156b (127.0.97.2:41425)
I20260926 21:37:04.120000 3463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:37:04.140969 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:04.458043 3791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:04.458374 3791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:04.458559 3791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:04.465827 3791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:04.465999 3791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:04.466183 3791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:04.475672 3791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:04.476994 3791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:04.478339 3791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:04.491255 3796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:04.491140 3799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:04.491175 3797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:04.492290 3791 server_base.cc:1034] running on GCE node
I20260926 21:37:04.492966 3791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:04.494011 3791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:04.495222 3791 hybrid_clock.cc:648] HybridClock initialized: now 1790458624495170 us; error 42 us; skew 500 ppm
I20260926 21:37:04.495642 3791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:04.498905 3791 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:37:04.500368 3791 webserver.cc:460] Webserver started at http://127.0.97.3:39437/ using document root <none> and password file <none>
I20260926 21:37:04.501049 3791 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:04.501240 3791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:04.501540 3791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:04.504326 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "cb6682c12764491b940480e8441accd7"
format_stamp: "Formatted at 2026-09-26 21:37:04 on dist-test-slave-lhhq"
server_key: "2107c60c027bb2ecf2642402e09e52ae"
server_key_iv: "038751971fbe877f59a9dbd0e8f4ab98"
server_key_version: "encryptionkey@0"
I20260926 21:37:04.505079 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cb6682c12764491b940480e8441accd7"
format_stamp: "Formatted at 2026-09-26 21:37:04 on dist-test-slave-lhhq"
server_key: "2107c60c027bb2ecf2642402e09e52ae"
server_key_iv: "038751971fbe877f59a9dbd0e8f4ab98"
server_key_version: "encryptionkey@0"
I20260926 21:37:04.510111 3791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:37:04.513542 3806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:04.514911 3791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:37:04.515076 3791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "cb6682c12764491b940480e8441accd7"
format_stamp: "Formatted at 2026-09-26 21:37:04 on dist-test-slave-lhhq"
server_key: "2107c60c027bb2ecf2642402e09e52ae"
server_key_iv: "038751971fbe877f59a9dbd0e8f4ab98"
server_key_version: "encryptionkey@0"
I20260926 21:37:04.515272 3791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:04.552776 3791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:04.557093 3791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:04.557417 3791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:04.558382 3791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:04.559698 3791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:04.559799 3791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:04.559921 3791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:04.560000 3791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:04.592332 3791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43779
I20260926 21:37:04.592351 3918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43779 every 8 connection(s)
I20260926 21:37:04.593925 3791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:04.599563 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 3791
I20260926 21:37:04.599768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:37:04.600217 388 external_mini_cluster.cc:1428] Setting key 0b2dec26285198c6d84e0e28cab47884
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:04.613329 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:04.596537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43885 (local address 127.0.97.62:45331)
0926 21:37:04.596707 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:04.596711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:04.598045 (+ 1334us) server_negotiation.cc:407] Connection header received
0926 21:37:04.599782 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:04.599790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:04.599883 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:04.600007 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:04.601716 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.602688 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.604276 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.604704 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.604809 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:04.608070 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:04.608075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:04.608078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:04.608110 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:04.610649 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.611298 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.611302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.611304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.611422 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.611795 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.611798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.611800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.612049 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:04.612160 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:04.612834 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 21:37:04.613003 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:37:04.614424 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:04.597049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45331 (local address 127.0.97.3:43885)
0926 21:37:04.597889 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:04.597930 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:04.599523 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:04.600223 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:04.600231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:04.600748 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:04.601566 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:04.601585 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.602825 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.602829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:04.604116 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:04.604128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.604889 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.604894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:04.605080 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:04.605795 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:04.605816 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:04.607879 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:04.610766 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:04.610773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:04.610785 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:04.611186 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:04.611529 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:04.611532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:04.611534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:04.611695 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:04.612167 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:04.612172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:04.612462 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 21:37:04.612680 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 21:37:04.613047 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":346,"threads_started":1}
I20260926 21:37:04.615917 3919 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45331
I20260926 21:37:04.616230 3919 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:04.617168 3919 heartbeater.cc:511] Master 127.0.97.62:45331 requested a full tablet report, sending...
I20260926 21:37:04.618783 3462 ts_manager.cc:194] Registered new tserver with Master: cb6682c12764491b940480e8441accd7 (127.0.97.3:43779)
I20260926 21:37:04.619800 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43885
I20260926 21:37:04.628108 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:37:04.630358 3652 heartbeater.cc:503] Master 127.0.97.62:45331 was elected leader, sending a full tablet report...
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458624, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458624, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:04.675225 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:04.660811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.0.97.62:45331)
0926 21:37:04.661023 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:04.661031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:04.661422 (+ 391us) server_negotiation.cc:407] Connection header received
0926 21:37:04.661664 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:04.661668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:04.661733 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:04.661841 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:04.663188 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.663898 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.665139 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.665613 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.665747 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:04.669870 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:04.669876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:04.669879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:04.669916 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:04.672338 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.673029 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.673036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.673040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.673174 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.673681 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.673687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.673691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.674027 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:04.674288 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:04.674632 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 21:37:04.674847 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 21:37:04.675839 3462 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60568
I20260926 21:37:04.677176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3524
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458624, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:37:04.692688 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:04.678427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.0.97.1:33331)
0926 21:37:04.679010 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:04.679018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:04.679041 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:37:04.679093 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:04.679099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:04.679316 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:04.679439 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:04.680464 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.681567 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.682719 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.683264 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.683357 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:04.687549 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:04.687559 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:04.687574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:04.687604 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:04.690132 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.690777 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.690792 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.690794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.690908 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.691269 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.691275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.691278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.691583 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:04.691881 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:04.692115 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 21:37:04.692364 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":302,"threads_started":1}
W20260926 21:37:04.694044 3647 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59872
W20260926 21:37:04.695118 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3524: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:37:04.695273 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 3524
I20260926 21:37:04.710902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3658
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458624, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:37:04.726816 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:04.711761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.0.97.2:41425)
0926 21:37:04.712329 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:04.712336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:04.712361 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:37:04.712417 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:04.712423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:04.712625 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:04.712759 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:04.714121 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.715235 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.716435 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.716965 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.717059 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:04.721357 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:04.721368 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:04.721382 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:04.721421 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:04.724162 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.724728 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.724735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.724737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.724850 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.725320 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.725323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.725325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.725745 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:04.725980 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:04.726269 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 21:37:04.726456 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":300,"threads_started":1}
W20260926 21:37:04.728319 3781 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41384
W20260926 21:37:04.729450 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3658: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:37:04.729642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 3658
I20260926 21:37:04.744999 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3791
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458624, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:37:04.761300 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:04.746168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.97.3:43779)
0926 21:37:04.746817 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:04.746824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:04.746852 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:37:04.746909 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:04.746915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:04.747162 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:04.747287 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:04.748679 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.749828 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.751018 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:04.751566 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:04.751661 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:04.755912 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:04.755922 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:04.755937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:04.755971 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:04.758759 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.759363 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.759369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.759372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.759483 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:04.759783 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:04.759786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:04.759788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:04.760148 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:04.760435 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:04.760652 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 21:37:04.760845 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":350,"threads_started":1}
W20260926 21:37:04.762620 3914 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57382
W20260926 21:37:04.763779 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3791: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:37:04.763936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 3791
I20260926 21:37:04.779147 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3431
I20260926 21:37:04.780375 3493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:04.881026 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 3431
2026-09-26T21:37:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3626 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3936](info): set up 2 sockets
Sep 26 21:37:04 dist-test-slave-lhhq krb5kdc[3936](info): commencing operation
krb5kdc: starting...
W20260926 21:37:06.013677 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:37:06 dist-test-slave-lhhq krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:37:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:37:06.209278 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44133
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:06.532647 3949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:06.533008 3949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:06.533177 3949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:06.540684 3949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:06.540840 3949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:06.540956 3949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:06.541052 3949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:06.541141 3949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:06.551385 3949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:06.552788 3949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:06.554214 3949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:06.566169 3957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:06.566160 3954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:06.566141 3955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:06.567394 3949 server_base.cc:1034] running on GCE node
I20260926 21:37:06.568300 3949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:06.569916 3949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:06.571177 3949 hybrid_clock.cc:648] HybridClock initialized: now 1790458626571132 us; error 51 us; skew 500 ppm
I20260926 21:37:06.571565 3949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:06 dist-test-slave-lhhq krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:06.583849 3949 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:37:06.585602 3949 webserver.cc:460] Webserver started at http://127.0.97.62:35055/ using document root <none> and password file <none>
I20260926 21:37:06.586341 3949 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:06.586495 3949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:06.586822 3949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:06.589890 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "7ec317a795a34690967c3ab217df881d"
format_stamp: "Formatted at 2026-09-26 21:37:06 on dist-test-slave-lhhq"
server_key: "2c1e1ddd7ab3de11d816e893c3c3600d"
server_key_iv: "656341a97b59001859adc8772e30c9c5"
server_key_version: "encryptionkey@0"
I20260926 21:37:06.590687 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "7ec317a795a34690967c3ab217df881d"
format_stamp: "Formatted at 2026-09-26 21:37:06 on dist-test-slave-lhhq"
server_key: "2c1e1ddd7ab3de11d816e893c3c3600d"
server_key_iv: "656341a97b59001859adc8772e30c9c5"
server_key_version: "encryptionkey@0"
I20260926 21:37:06.595947 3949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:37:06.599535 3964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:06.601028 3949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:37:06.601296 3949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "7ec317a795a34690967c3ab217df881d"
format_stamp: "Formatted at 2026-09-26 21:37:06 on dist-test-slave-lhhq"
server_key: "2c1e1ddd7ab3de11d816e893c3c3600d"
server_key_iv: "656341a97b59001859adc8772e30c9c5"
server_key_version: "encryptionkey@0"
I20260926 21:37:06.601548 3949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:06.625150 3949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:06.629670 3949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:06.629973 3949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:06.648663 3949 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44133
I20260926 21:37:06.648671 4015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44133 every 8 connection(s)
I20260926 21:37:06.650151 3949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:06.653914 4016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:06.658371 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 3949
I20260926 21:37:06.658573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:06.659389 388 external_mini_cluster.cc:1428] Setting key 063437f75099f43bf23cc2b9e9e94a27
I20260926 21:37:06.665863 4016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d: Bootstrap starting.
I20260926 21:37:06.669662 4016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:06.670892 4016 log.cc:824] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:06.674510 4016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d: No bootstrap required, opened a new log
Sep 26 21:37:06 dist-test-slave-lhhq krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:06.685403 4016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec317a795a34690967c3ab217df881d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44133 } }
I20260926 21:37:06.685861 4016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:06.685966 4016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec317a795a34690967c3ab217df881d, State: Initialized, Role: FOLLOWER
I20260926 21:37:06.686136 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:06.661619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.0.97.62:44133)
0926 21:37:06.662485 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:06.662498 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:06.662551 (+ 53us) server_negotiation.cc:407] Connection header received
0926 21:37:06.664587 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:06.664612 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:06.665052 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:06.665571 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:06.667389 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:06.668734 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:06.670519 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:06.671257 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:06.671495 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:06.676798 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:06.676810 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:06.676836 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:06.676895 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:06.681124 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:06.681994 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:06.682004 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:06.682014 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:06.682235 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:06.682628 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:06.682633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:06.682637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:06.683470 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:06.683824 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:06.684214 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 21:37:06.684696 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":399,"threads_started":1}
I20260926 21:37:06.686722 4016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec317a795a34690967c3ab217df881d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44133 } }
I20260926 21:37:06.686935 4016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:06.687093 4016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:06.687275 4016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:06.689719 4016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec317a795a34690967c3ab217df881d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44133 } }
I20260926 21:37:06.690213 4016 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec317a795a34690967c3ab217df881d; no voters:
I20260926 21:37:06.690788 4016 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:06.690961 4021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:06.691658 4021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [term 1 LEADER]: Becoming Leader. State: Replica: 7ec317a795a34690967c3ab217df881d, State: Running, Role: LEADER
I20260926 21:37:06.692217 4021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec317a795a34690967c3ab217df881d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44133 } }
I20260926 21:37:06.692826 4016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:06.695580 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec317a795a34690967c3ab217df881d. Latest consensus state: current_term: 1 leader_uuid: "7ec317a795a34690967c3ab217df881d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec317a795a34690967c3ab217df881d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44133 } } }
I20260926 21:37:06.695971 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:06.696632 4022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ec317a795a34690967c3ab217df881d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec317a795a34690967c3ab217df881d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44133 } } }
I20260926 21:37:06.697101 4022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:06.698063 4028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:06.711187 4028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:06.722818 4028 catalog_manager.cc:1347] Generated new cluster ID: 0305dafef10e44ecb00dfe47759f8aad
I20260926 21:37:06.722918 4028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:06.739077 4028 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:06.740481 4028 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:06.748014 4028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ec317a795a34690967c3ab217df881d: Generated new TSK 0
I20260926 21:37:06.748901 4028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:37:06.808503 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44133
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:07.133986 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:07.134265 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:07.134358 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:07.141350 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:07.141448 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:07.141556 4043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:07.150830 4043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:07.151994 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:07.153215 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:07.166775 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:07.166815 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:07.166990 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:07.167446 4043 server_base.cc:1034] running on GCE node
I20260926 21:37:07.168141 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:07.169056 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:07.170251 4043 hybrid_clock.cc:648] HybridClock initialized: now 1790458627170171 us; error 74 us; skew 500 ppm
I20260926 21:37:07.170526 4043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:07 dist-test-slave-lhhq krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:07.173738 4043 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:37:07.175141 4043 webserver.cc:460] Webserver started at http://127.0.97.1:42323/ using document root <none> and password file <none>
I20260926 21:37:07.175758 4043 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:07.175853 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:07.176091 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:07.179355 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "3719a75d6e744ec285aa3421df8accd3"
format_stamp: "Formatted at 2026-09-26 21:37:07 on dist-test-slave-lhhq"
server_key: "e6098780179372907189e51acc9d49b4"
server_key_iv: "3415afd42ad0b8d63a836cc0c83b3379"
server_key_version: "encryptionkey@0"
I20260926 21:37:07.180071 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3719a75d6e744ec285aa3421df8accd3"
format_stamp: "Formatted at 2026-09-26 21:37:07 on dist-test-slave-lhhq"
server_key: "e6098780179372907189e51acc9d49b4"
server_key_iv: "3415afd42ad0b8d63a836cc0c83b3379"
server_key_version: "encryptionkey@0"
I20260926 21:37:07.184852 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260926 21:37:07.188158 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:07.189497 4043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:07.189682 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "3719a75d6e744ec285aa3421df8accd3"
format_stamp: "Formatted at 2026-09-26 21:37:07 on dist-test-slave-lhhq"
server_key: "e6098780179372907189e51acc9d49b4"
server_key_iv: "3415afd42ad0b8d63a836cc0c83b3379"
server_key_version: "encryptionkey@0"
I20260926 21:37:07.189862 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:07.211305 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:07.215570 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:07.215853 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:07.216795 4043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:07.218230 4043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:07.218338 4043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:07.218488 4043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:07.218550 4043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:07.251737 4043 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41605
I20260926 21:37:07.251789 4170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41605 every 8 connection(s)
I20260926 21:37:07.252966 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:07.257870 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 4043
I20260926 21:37:07.258071 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:37:07.258678 388 external_mini_cluster.cc:1428] Setting key cc23adaa3db958ba5ba3cf30e6b7639e
Sep 26 21:37:07 dist-test-slave-lhhq krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:37:07.280412 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:07.255662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38453 (local address 127.0.97.62:44133)
0926 21:37:07.256112 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:07.256119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:07.256926 (+ 807us) server_negotiation.cc:407] Connection header received
0926 21:37:07.258904 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:07.258910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:07.258990 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:07.259116 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:07.260783 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:07.261563 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:07.263067 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:07.263471 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:07.263607 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:07.274563 (+ 10956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:07.274569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:07.274573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:07.274610 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:07.277235 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:07.277988 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:07.277992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:07.277994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:07.278113 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:07.278547 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:07.278550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:07.278553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:07.278830 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:07.278952 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:07.279833 (+ 881us) server_negotiation.cc:299] Negotiation successful
0926 21:37:07.280049 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":250,"threads_started":1}
I20260926 21:37:07.281507 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:07.256090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44133 (local address 127.0.97.1:38453)
0926 21:37:07.256769 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:07.256806 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:07.258599 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:07.259366 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:07.259374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:07.259982 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:07.260623 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:07.260640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:07.261695 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:07.261702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:07.262922 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:07.262933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:07.263698 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:07.263706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:07.263933 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:07.270942 (+ 7009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:07.270985 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:07.274250 (+ 3265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:07.277435 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:07.277443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:07.277458 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:07.277880 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:07.278223 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:07.278226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:07.278228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:07.278452 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:07.278969 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:07.278981 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:07.279459 (+ 478us) client_negotiation.cc:769] Sending connection context
0926 21:37:07.279736 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 21:37:07.280292 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":267,"threads_started":1}
I20260926 21:37:07.283269 4171 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44133
I20260926 21:37:07.283576 4171 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:07.284294 4171 heartbeater.cc:511] Master 127.0.97.62:44133 requested a full tablet report, sending...
I20260926 21:37:07.286659 3981 ts_manager.cc:194] Registered new tserver with Master: 3719a75d6e744ec285aa3421df8accd3 (127.0.97.1:41605)
I20260926 21:37:07.288816 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:37:07.305274 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44133
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:07.636742 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:07.637035 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:07.637252 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:07.644512 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:07.644630 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:07.644845 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:07.654727 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:07.656008 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:07.657394 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:07.670400 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:07.670363 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:07.670399 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:07.671217 4177 server_base.cc:1034] running on GCE node
I20260926 21:37:07.671851 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:07.672816 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:07.674125 4177 hybrid_clock.cc:648] HybridClock initialized: now 1790458627674056 us; error 60 us; skew 500 ppm
I20260926 21:37:07.674482 4177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:07 dist-test-slave-lhhq krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:07.677618 4177 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:37:07.679137 4177 webserver.cc:460] Webserver started at http://127.0.97.2:43237/ using document root <none> and password file <none>
I20260926 21:37:07.679867 4177 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:07.680017 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:07.680334 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:07.683341 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "af20a046241643299c0ffeea19229568"
format_stamp: "Formatted at 2026-09-26 21:37:07 on dist-test-slave-lhhq"
server_key: "07b0875f5bc431169810bb4ac2f96b0c"
server_key_iv: "053afc77fb94a2ecbb4cbc4d3433f431"
server_key_version: "encryptionkey@0"
I20260926 21:37:07.684156 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "af20a046241643299c0ffeea19229568"
format_stamp: "Formatted at 2026-09-26 21:37:07 on dist-test-slave-lhhq"
server_key: "07b0875f5bc431169810bb4ac2f96b0c"
server_key_iv: "053afc77fb94a2ecbb4cbc4d3433f431"
server_key_version: "encryptionkey@0"
I20260926 21:37:07.689661 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 21:37:07.693264 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:07.694912 4177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:07.695123 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "af20a046241643299c0ffeea19229568"
format_stamp: "Formatted at 2026-09-26 21:37:07 on dist-test-slave-lhhq"
server_key: "07b0875f5bc431169810bb4ac2f96b0c"
server_key_iv: "053afc77fb94a2ecbb4cbc4d3433f431"
server_key_version: "encryptionkey@0"
I20260926 21:37:07.695366 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:07.719641 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:07.725879 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:07.726244 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:07.727538 4177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:07.729171 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:07.729305 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 21:37:07.729432 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:07.729514 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:07.764199 4177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35045
I20260926 21:37:07.764242 4304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35045 every 8 connection(s)
I20260926 21:37:07.765678 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:07.774031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 4177
I20260926 21:37:07.774227 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:37:07.774659 388 external_mini_cluster.cc:1428] Setting key 2d9aad7571ee1b3cb23a9160e8d34126
Sep 26 21:37:07 dist-test-slave-lhhq krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:07.787791 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:07.768460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52799 (local address 127.0.97.62:44133)
0926 21:37:07.768636 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:07.768640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:07.770072 (+ 1432us) server_negotiation.cc:407] Connection header received
0926 21:37:07.771922 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:07.771928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:07.772023 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:07.772134 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:07.773801 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:07.774711 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:07.776686 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:07.777298 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:07.777407 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:07.781960 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:07.781967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:07.781971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:07.782012 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:07.784666 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:07.785522 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:07.785526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:07.785528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:07.785656 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:07.786087 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:07.786092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:07.786102 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:07.786356 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:07.786517 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:07.787208 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 21:37:07.787424 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:37:07.788833 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:07.769073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44133 (local address 127.0.97.2:52799)
0926 21:37:07.769898 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:07.769942 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:07.771630 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:07.772387 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:07.772395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:07.772933 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:07.773589 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:07.773608 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:07.774902 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:07.774907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:07.776528 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:07.776545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:07.777542 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:07.777550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:07.777886 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:07.778778 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:07.778813 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:07.781680 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:07.784823 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:07.784832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:07.784850 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:07.785386 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:07.785770 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:07.785775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:07.785779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:07.785968 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:07.786489 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:07.786497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:07.786848 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 21:37:07.787126 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 21:37:07.787687 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":308,"threads_started":1}
I20260926 21:37:07.790628 4305 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44133
I20260926 21:37:07.790943 4305 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:07.791700 4305 heartbeater.cc:511] Master 127.0.97.62:44133 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:37:07.793349 3981 ts_manager.cc:194] Registered new tserver with Master: af20a046241643299c0ffeea19229568 (127.0.97.2:35045)
I20260926 21:37:07.794277 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:37:07.820542 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44133
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:08.145237 4310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:08.145558 4310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:08.145727 4310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:08.152884 4310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:08.153041 4310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:08.153246 4310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:08.164438 4310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:08.165840 4310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:08.167204 4310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:08.180137 4316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:08.180114 4318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:08.180343 4315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:08.180853 4310 server_base.cc:1034] running on GCE node
I20260926 21:37:08.181504 4310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:08.182401 4310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:08.183624 4310 hybrid_clock.cc:648] HybridClock initialized: now 1790458628183581 us; error 54 us; skew 500 ppm
I20260926 21:37:08.183943 4310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:08.187021 4310 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:37:08.188419 4310 webserver.cc:460] Webserver started at http://127.0.97.3:33557/ using document root <none> and password file <none>
I20260926 21:37:08.189088 4310 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:08.189255 4310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:08.189584 4310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:08.192364 4310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "a575da5d18aa45cb846b5414c87fe949"
format_stamp: "Formatted at 2026-09-26 21:37:08 on dist-test-slave-lhhq"
server_key: "6f779092ae7cc5ef552af564ed7f70c4"
server_key_iv: "1b8b59e6840ef7d6475f53b2fe7c72d2"
server_key_version: "encryptionkey@0"
I20260926 21:37:08.193282 4310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a575da5d18aa45cb846b5414c87fe949"
format_stamp: "Formatted at 2026-09-26 21:37:08 on dist-test-slave-lhhq"
server_key: "6f779092ae7cc5ef552af564ed7f70c4"
server_key_iv: "1b8b59e6840ef7d6475f53b2fe7c72d2"
server_key_version: "encryptionkey@0"
I20260926 21:37:08.198554 4310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:37:08.202015 4325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:08.203325 4310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:37:08.203500 4310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "a575da5d18aa45cb846b5414c87fe949"
format_stamp: "Formatted at 2026-09-26 21:37:08 on dist-test-slave-lhhq"
server_key: "6f779092ae7cc5ef552af564ed7f70c4"
server_key_iv: "1b8b59e6840ef7d6475f53b2fe7c72d2"
server_key_version: "encryptionkey@0"
I20260926 21:37:08.203673 4310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:08.217527 4310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:08.221629 4310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:08.221936 4310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:08.222913 4310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:08.224332 4310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:08.224407 4310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:08.224536 4310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:08.224597 4310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:08.258981 4310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43695
I20260926 21:37:08.259022 4437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43695 every 8 connection(s)
I20260926 21:37:08.260272 4310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:08.270356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 4310
I20260926 21:37:08.270566 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:37:08.271129 388 external_mini_cluster.cc:1428] Setting key 455dbab88456efc57f00df4ec7555aee
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:08.280915 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:08.262834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53177 (local address 127.0.97.62:44133)
0926 21:37:08.263050 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:08.263056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:08.264311 (+ 1255us) server_negotiation.cc:407] Connection header received
0926 21:37:08.266131 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:08.266136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:08.266203 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:08.266295 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:08.267973 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.268793 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.270498 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.271114 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.271262 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:08.275313 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:08.275318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:08.275322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:08.275357 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:08.277948 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.278727 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.278731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.278734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.278857 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.279325 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.279328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.279330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.279580 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:08.279682 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:08.280482 (+ 800us) server_negotiation.cc:299] Negotiation successful
0926 21:37:08.280653 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:37:08.282050 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:08.263337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44133 (local address 127.0.97.3:53177)
0926 21:37:08.264115 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:08.264165 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:08.265859 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:08.266514 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:08.266523 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:08.267056 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:08.267785 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:08.267807 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.268942 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.268945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:08.270312 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:08.270324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.271382 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.271387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:08.271596 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:08.272379 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:08.272415 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:08.275118 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:08.278120 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:08.278128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:08.278144 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:08.278593 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:08.278977 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:08.278981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:08.278983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:08.279202 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:08.279692 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:08.279699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:08.280137 (+ 438us) client_negotiation.cc:769] Sending connection context
0926 21:37:08.280377 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 21:37:08.280805 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":298,"threads_started":1}
I20260926 21:37:08.283682 4438 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44133
I20260926 21:37:08.284008 4438 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:08.284906 4438 heartbeater.cc:511] Master 127.0.97.62:44133 requested a full tablet report, sending...
I20260926 21:37:08.286845 3980 ts_manager.cc:194] Registered new tserver with Master: a575da5d18aa45cb846b5414c87fe949 (127.0.97.3:43695)
I20260926 21:37:08.287689 3980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53177
I20260926 21:37:08.292781 4171 heartbeater.cc:503] Master 127.0.97.62:44133 was elected leader, sending a full tablet report...
I20260926 21:37:08.299170 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:37:08.346925 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:08.331856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.0.97.1:41605)
0926 21:37:08.332524 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:08.332532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:08.332559 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:37:08.332619 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:08.332625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:08.332839 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:08.332983 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:08.334634 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.335633 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.336863 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.337531 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.337637 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:08.341435 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:08.341446 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:08.341463 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:08.341498 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:08.344311 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.344937 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.344944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.344946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.345060 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.345595 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.345599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.345602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.345931 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:08.346170 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:08.346457 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 21:37:08.346644 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":351,"threads_started":1}
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:08.364493 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:08.351180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.0.97.62:44133)
0926 21:37:08.351431 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:08.351437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:08.351668 (+ 231us) server_negotiation.cc:407] Connection header received
0926 21:37:08.351870 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:08.351874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:08.351934 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:08.352034 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:08.353241 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.353936 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.355158 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.355548 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.355675 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:08.359527 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:08.359533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:08.359536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:08.359569 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:08.361963 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.362580 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.362586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.362590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.362718 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.363061 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.363067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.363080 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.363383 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:08.363634 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:08.363954 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 21:37:08.364182 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 21:37:08.365057 3980 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54950
I20260926 21:37:08.366109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 4043
I20260926 21:37:08.378731 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:08.367020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.0.97.1:41605)
0926 21:37:08.367227 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:08.367235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:08.367379 (+ 144us) server_negotiation.cc:407] Connection header received
0926 21:37:08.367495 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:08.367499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:08.367587 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:08.367743 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:08.368890 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.369645 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.370807 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.371220 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.371315 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:08.373543 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:08.373548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:08.373551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:08.373583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:08.376173 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.376888 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.376891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.376894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.377013 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.377499 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.377502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.377504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.377768 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:08.377950 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:08.378211 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 21:37:08.378371 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:37:08.379390 4166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:08.462473 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 4043
I20260926 21:37:08.478925 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 4177
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:37:08.496027 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:08.479930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.0.97.2:35045)
0926 21:37:08.480586 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:08.480594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:08.480646 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:37:08.480707 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:08.480714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:08.480936 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:08.481071 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:08.482386 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.483446 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.484841 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.485557 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.485698 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:08.490132 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:08.490146 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:08.490163 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:08.490216 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:08.493378 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.494128 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.494136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.494138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.494265 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.494671 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.494675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.494677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.495025 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:08.495265 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:08.495509 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 21:37:08.495729 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":353,"threads_started":1}
I20260926 21:37:08.497416 4300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:08.578725 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 4177
I20260926 21:37:08.594770 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 4310
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:37:08.611347 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:08.595896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.0.97.3:43695)
0926 21:37:08.596474 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:08.596481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:08.596509 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:37:08.596568 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:08.596574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:08.596794 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:08.596917 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:08.598163 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.599161 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.600379 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:08.600921 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:08.601016 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:08.605471 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:08.605484 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:08.605501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:08.605537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:08.608435 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.609128 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.609134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.609137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.609310 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:08.609683 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:08.609687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:08.609689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:08.610088 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:08.610362 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:08.610656 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 21:37:08.611006 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":290,"threads_started":1}
I20260926 21:37:08.612644 4433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:08.701471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 4310
I20260926 21:37:08.717094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 3949
I20260926 21:37:08.718278 4011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:08.820747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 3949
2026-09-26T21:37:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3938 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[4457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[4457](info): set up 2 sockets
Sep 26 21:37:08 dist-test-slave-lhhq krb5kdc[4457](info): commencing operation
krb5kdc: starting...
W20260926 21:37:09.909670 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:37:10 dist-test-slave-lhhq krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:37:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:37:10.106784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35953
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:10.423880 4470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:10.424170 4470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:10.424263 4470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:10.431516 4470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:10.431648 4470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:10.431694 4470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:10.431744 4470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:10.431800 4470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:10.441810 4470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:10.443085 4470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:10.444356 4470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:10.455655 4475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:10.455629 4476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:10.455524 4478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:10.456295 4470 server_base.cc:1034] running on GCE node
I20260926 21:37:10.457044 4470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:10.458402 4470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:10.459645 4470 hybrid_clock.cc:648] HybridClock initialized: now 1790458630459607 us; error 51 us; skew 500 ppm
I20260926 21:37:10.459990 4470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:10 dist-test-slave-lhhq krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:10.463420 4470 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:37:10.464522 4470 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:10.464612 4470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:10.464872 4470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:10.467773 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "ff66139901bf44dd989745bdb31b661f"
format_stamp: "Formatted at 2026-09-26 21:37:10 on dist-test-slave-lhhq"
server_key: "bf9c41b4ba09771ac4addc073ad72970"
server_key_iv: "364f84d16b3abf1eae3dc5a26e75c532"
server_key_version: "encryptionkey@0"
I20260926 21:37:10.468488 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff66139901bf44dd989745bdb31b661f"
format_stamp: "Formatted at 2026-09-26 21:37:10 on dist-test-slave-lhhq"
server_key: "bf9c41b4ba09771ac4addc073ad72970"
server_key_iv: "364f84d16b3abf1eae3dc5a26e75c532"
server_key_version: "encryptionkey@0"
I20260926 21:37:10.473662 4470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:37:10.476976 4484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:10.478327 4470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:10.478506 4470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "ff66139901bf44dd989745bdb31b661f"
format_stamp: "Formatted at 2026-09-26 21:37:10 on dist-test-slave-lhhq"
server_key: "bf9c41b4ba09771ac4addc073ad72970"
server_key_iv: "364f84d16b3abf1eae3dc5a26e75c532"
server_key_version: "encryptionkey@0"
I20260926 21:37:10.478735 4470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:10.500675 4470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:10.504848 4470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:10.505103 4470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:10.523058 4470 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35953
I20260926 21:37:10.523084 4535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35953 every 8 connection(s)
I20260926 21:37:10.523962 4470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:10.524394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 4470
I20260926 21:37:10.524626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:10.525331 388 external_mini_cluster.cc:1428] Setting key 95b66b9e90235d30ee87f62d10fd035a
I20260926 21:37:10.528237 4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:37:10 dist-test-slave-lhhq krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:10.541319 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f: Bootstrap starting.
I20260926 21:37:10.544358 4536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:10.545466 4536 log.cc:824] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:10.548491 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f: No bootstrap required, opened a new log
I20260926 21:37:10.548741 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:10.527504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.0.97.62:35953)
0926 21:37:10.528379 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:10.528394 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:10.528437 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:37:10.530167 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:10.530188 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:10.530586 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:10.530942 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:10.532727 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:10.534177 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:10.535974 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:10.536649 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:10.536778 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:10.541501 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:10.541509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:10.541526 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:10.541562 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:10.544530 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:10.545210 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:10.545218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:10.545232 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:10.545386 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:10.545798 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:10.545804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:10.545807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:10.546622 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:10.546872 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:10.547319 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 21:37:10.547715 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":376,"threads_started":1}
I20260926 21:37:10.555732 4536 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff66139901bf44dd989745bdb31b661f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35953 } }
I20260926 21:37:10.556128 4536 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:10.556236 4536 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff66139901bf44dd989745bdb31b661f, State: Initialized, Role: FOLLOWER
I20260926 21:37:10.556787 4536 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff66139901bf44dd989745bdb31b661f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35953 } }
I20260926 21:37:10.557020 4536 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:10.557155 4536 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:10.557346 4536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:10.559782 4536 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff66139901bf44dd989745bdb31b661f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35953 } }
I20260926 21:37:10.560235 4536 leader_election.cc:302] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff66139901bf44dd989745bdb31b661f; no voters:
I20260926 21:37:10.560868 4536 leader_election.cc:288] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:10.561049 4541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:10.561860 4541 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [term 1 LEADER]: Becoming Leader. State: Replica: ff66139901bf44dd989745bdb31b661f, State: Running, Role: LEADER
I20260926 21:37:10.562505 4541 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff66139901bf44dd989745bdb31b661f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35953 } }
I20260926 21:37:10.563056 4536 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:10.565477 4543 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff66139901bf44dd989745bdb31b661f. Latest consensus state: current_term: 1 leader_uuid: "ff66139901bf44dd989745bdb31b661f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff66139901bf44dd989745bdb31b661f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35953 } } }
I20260926 21:37:10.565826 4543 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:10.566282 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff66139901bf44dd989745bdb31b661f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff66139901bf44dd989745bdb31b661f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35953 } } }
I20260926 21:37:10.566617 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:10.567427 4548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:10.572347 4548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:10.581013 4548 catalog_manager.cc:1347] Generated new cluster ID: 7fac4531cb3440eb85d73e2e0296485a
I20260926 21:37:10.581141 4548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:10.593464 4548 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:10.594789 4548 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:10.604506 4548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f: Generated new TSK 0
I20260926 21:37:10.605408 4548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:37:10.662205 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35953
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:10.999706 4562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:10.999984 4562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:11.000150 4562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:11.007437 4562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:11.007552 4562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:11.007751 4562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:11.017596 4562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:11.018899 4562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:11.020311 4562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:11.033083 4568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:11.033241 4570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:11.033082 4567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:11.034001 4562 server_base.cc:1034] running on GCE node
I20260926 21:37:11.034628 4562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:11.035614 4562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:11.036851 4562 hybrid_clock.cc:648] HybridClock initialized: now 1790458631036792 us; error 54 us; skew 500 ppm
I20260926 21:37:11.037225 4562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:11 dist-test-slave-lhhq krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:11.040403 4562 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:37:11.041675 4562 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:11.041805 4562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:11.042119 4562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:11.044999 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "edefeaee572842b89b2b67b1adc46248"
format_stamp: "Formatted at 2026-09-26 21:37:11 on dist-test-slave-lhhq"
server_key: "67f5c757cd7e34f2eba9f0d31ad50249"
server_key_iv: "2615a1a3c088e7a0831b916e6bae7b33"
server_key_version: "encryptionkey@0"
I20260926 21:37:11.045864 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "edefeaee572842b89b2b67b1adc46248"
format_stamp: "Formatted at 2026-09-26 21:37:11 on dist-test-slave-lhhq"
server_key: "67f5c757cd7e34f2eba9f0d31ad50249"
server_key_iv: "2615a1a3c088e7a0831b916e6bae7b33"
server_key_version: "encryptionkey@0"
I20260926 21:37:11.051345 4562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:37:11.054821 4576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:11.056162 4562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:37:11.056330 4562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "edefeaee572842b89b2b67b1adc46248"
format_stamp: "Formatted at 2026-09-26 21:37:11 on dist-test-slave-lhhq"
server_key: "67f5c757cd7e34f2eba9f0d31ad50249"
server_key_iv: "2615a1a3c088e7a0831b916e6bae7b33"
server_key_version: "encryptionkey@0"
I20260926 21:37:11.056542 4562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:11.074023 4562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:11.077916 4562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:11.078192 4562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:11.079113 4562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:11.080440 4562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:11.080559 4562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:11.080709 4562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:11.080799 4562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:11.112813 4562 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40421
I20260926 21:37:11.112859 4688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40421 every 8 connection(s)
I20260926 21:37:11.113727 4562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:11.121922 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 4562
I20260926 21:37:11.122133 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:37:11.122804 388 external_mini_cluster.cc:1428] Setting key 4ddfed7de7541ed8c183daf930ff2863
Sep 26 21:37:11 dist-test-slave-lhhq krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:11.134335 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:11.116088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51991 (local address 127.0.97.62:35953)
0926 21:37:11.116546 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:11.116553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:11.117564 (+ 1011us) server_negotiation.cc:407] Connection header received
0926 21:37:11.119272 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:11.119277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:11.119363 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:11.119469 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:11.120873 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:11.121706 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:11.124233 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:11.124659 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:11.124799 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:11.128560 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:11.128565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:11.128568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:11.128600 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:11.131232 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:11.131958 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:11.131964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:11.131968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:11.132087 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:11.132522 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:11.132527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:11.132530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:11.132808 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:11.132961 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:11.133747 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 21:37:11.133964 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":215,"threads_started":1}
I20260926 21:37:11.135308 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:11.116684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35953 (local address 127.0.97.1:51991)
0926 21:37:11.117409 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:11.117455 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:11.119018 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:11.119656 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:11.119664 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:11.120147 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:11.120701 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:11.120719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:11.121888 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:11.121892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:11.123944 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:11.123960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:11.124896 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:11.124903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:11.125141 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:11.125913 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:11.125940 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:11.128321 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:11.131379 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:11.131388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:11.131407 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:11.131833 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:11.132203 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:11.132209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:11.132213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:11.132398 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:11.132940 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:11.132948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:11.133345 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 21:37:11.133618 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 21:37:11.134159 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":275,"threads_started":1}
I20260926 21:37:11.136986 4689 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35953
I20260926 21:37:11.137425 4689 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:11.138479 4689 heartbeater.cc:511] Master 127.0.97.62:35953 requested a full tablet report, sending...
I20260926 21:37:11.141347 4501 ts_manager.cc:194] Registered new tserver with Master: edefeaee572842b89b2b67b1adc46248 (127.0.97.1:40421)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:37:11.143464 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:37:11.167977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35953
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:11.496657 4695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:11.496946 4695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:11.497045 4695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:11.504261 4695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:11.504374 4695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:11.504489 4695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:11.514374 4695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:11.516073 4695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:11.517374 4695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:11.529561 4700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:11.531345 4701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:11.531635 4703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:11.532336 4695 server_base.cc:1034] running on GCE node
I20260926 21:37:11.532992 4695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:11.534049 4695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:11.535286 4695 hybrid_clock.cc:648] HybridClock initialized: now 1790458631535236 us; error 42 us; skew 500 ppm
I20260926 21:37:11.535614 4695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:11 dist-test-slave-lhhq krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:11.539083 4695 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:37:11.540463 4695 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:11.540570 4695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:11.540903 4695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:11.544715 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "ff3d34e544354e60b252ef0a002937b1"
format_stamp: "Formatted at 2026-09-26 21:37:11 on dist-test-slave-lhhq"
server_key: "3372206bf7ffb857e1d2d08c5a5af289"
server_key_iv: "8ef6cb02f3c5b1a6f78ae2c10231ebf2"
server_key_version: "encryptionkey@0"
I20260926 21:37:11.545531 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ff3d34e544354e60b252ef0a002937b1"
format_stamp: "Formatted at 2026-09-26 21:37:11 on dist-test-slave-lhhq"
server_key: "3372206bf7ffb857e1d2d08c5a5af289"
server_key_iv: "8ef6cb02f3c5b1a6f78ae2c10231ebf2"
server_key_version: "encryptionkey@0"
I20260926 21:37:11.550361 4695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:37:11.553598 4709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:11.554975 4695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:37:11.555155 4695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "ff3d34e544354e60b252ef0a002937b1"
format_stamp: "Formatted at 2026-09-26 21:37:11 on dist-test-slave-lhhq"
server_key: "3372206bf7ffb857e1d2d08c5a5af289"
server_key_iv: "8ef6cb02f3c5b1a6f78ae2c10231ebf2"
server_key_version: "encryptionkey@0"
I20260926 21:37:11.555363 4695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:11.577154 4695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:11.581045 4695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:11.581334 4695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:11.582325 4695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:11.583720 4695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:11.583841 4695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:11.583974 4695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:11.584061 4695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:11.618023 4695 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41839
I20260926 21:37:11.618117 4821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41839 every 8 connection(s)
I20260926 21:37:11.618901 4695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:11.620143 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 4695
I20260926 21:37:11.620325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:37:11.620813 388 external_mini_cluster.cc:1428] Setting key 19580a41ddd5927dcbf8faa67070d8a3
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Sep 26 21:37:11 dist-test-slave-lhhq krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:11.650858 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:11.621588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40155 (local address 127.0.97.62:35953)
0926 21:37:11.621801 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:11.621808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:11.623143 (+ 1335us) server_negotiation.cc:407] Connection header received
0926 21:37:11.625082 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:11.625091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:11.625200 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:11.625313 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:11.626747 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:11.627856 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:11.629452 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:11.630214 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:11.630354 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:11.643913 (+ 13559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:11.643923 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:11.643930 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:11.643984 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:11.647408 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:11.648203 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:11.648212 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:11.648218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:11.648389 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:11.648835 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:11.648843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:11.648849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:11.649298 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:11.649455 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:11.650153 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 21:37:11.650416 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 21:37:11.651674 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:11.622114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35953 (local address 127.0.97.2:40155)
0926 21:37:11.622966 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:11.623013 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:11.624801 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:11.625498 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:11.625507 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:11.626005 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:11.626555 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:11.626572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:11.627997 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:11.628002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:11.629314 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:11.629325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:11.630423 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:11.630428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:11.630615 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:11.640971 (+ 10356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:11.641001 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:11.643578 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:11.647583 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:11.647593 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:11.647612 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:11.648066 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:11.648508 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:11.648512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:11.648514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:11.648718 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:11.649416 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:11.649422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:11.649781 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 21:37:11.650019 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 21:37:11.650419 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":392,"threads_started":1}
I20260926 21:37:11.653252 4822 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35953
I20260926 21:37:11.653551 4822 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:11.654184 4822 heartbeater.cc:511] Master 127.0.97.62:35953 requested a full tablet report, sending...
I20260926 21:37:11.655632 4501 ts_manager.cc:194] Registered new tserver with Master: ff3d34e544354e60b252ef0a002937b1 (127.0.97.2:41839)
I20260926 21:37:11.656455 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:37:11.668476 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35953
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:11.994948 4827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:11.995302 4827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:11.995491 4827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:12.002863 4827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:12.003041 4827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:12.003242 4827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:12.013140 4827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:12.014578 4827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:12.015918 4827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:12.029281 4832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:12.029201 4835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:12.029255 4833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:12.030162 4827 server_base.cc:1034] running on GCE node
I20260926 21:37:12.030715 4827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:12.031668 4827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:12.032866 4827 hybrid_clock.cc:648] HybridClock initialized: now 1790458632032821 us; error 35 us; skew 500 ppm
I20260926 21:37:12.033244 4827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:12 dist-test-slave-lhhq krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:12.036383 4827 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:37:12.037617 4827 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:12.037741 4827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:12.038071 4827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:12.041005 4827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "e38b2f155e1d41b19551254f787431d5"
format_stamp: "Formatted at 2026-09-26 21:37:12 on dist-test-slave-lhhq"
server_key: "ec51c57fc1796d4ccb0288d5ac2a8d13"
server_key_iv: "d54611b70cc1eb6f3b4a5028eb6d1f11"
server_key_version: "encryptionkey@0"
I20260926 21:37:12.041923 4827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e38b2f155e1d41b19551254f787431d5"
format_stamp: "Formatted at 2026-09-26 21:37:12 on dist-test-slave-lhhq"
server_key: "ec51c57fc1796d4ccb0288d5ac2a8d13"
server_key_iv: "d54611b70cc1eb6f3b4a5028eb6d1f11"
server_key_version: "encryptionkey@0"
I20260926 21:37:12.046932 4827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:37:12.050261 4841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:12.051553 4827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:12.051762 4827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "e38b2f155e1d41b19551254f787431d5"
format_stamp: "Formatted at 2026-09-26 21:37:12 on dist-test-slave-lhhq"
server_key: "ec51c57fc1796d4ccb0288d5ac2a8d13"
server_key_iv: "d54611b70cc1eb6f3b4a5028eb6d1f11"
server_key_version: "encryptionkey@0"
I20260926 21:37:12.052094 4827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:12.073850 4827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:12.077987 4827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:12.078266 4827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:12.079231 4827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:12.080515 4827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:12.080638 4827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:12.080773 4827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:12.080861 4827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:12.113622 4827 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43421
I20260926 21:37:12.113727 4953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43421 every 8 connection(s)
I20260926 21:37:12.114475 4827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:12.116567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 4827
I20260926 21:37:12.116741 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:37:12.117316 388 external_mini_cluster.cc:1428] Setting key c67bef55eb534766e128a2ff8600a739
Sep 26 21:37:12 dist-test-slave-lhhq krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:12.134253 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.117097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35749 (local address 127.0.97.62:35953)
0926 21:37:12.117309 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.117316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.118891 (+ 1575us) server_negotiation.cc:407] Connection header received
0926 21:37:12.121030 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.121034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.121095 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.121221 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:12.122618 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.123375 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.124882 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.125323 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.125422 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.128558 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:12.128563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:12.128566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:12.128595 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:12.131345 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.132032 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.132038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.132041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.132170 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.132563 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.132570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.132573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.132836 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:12.132987 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.133716 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.133923 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:37:12.135182 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.117729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35953 (local address 127.0.97.3:35749)
0926 21:37:12.118721 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:12.118767 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:12.120796 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:12.121416 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:12.121425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:12.121923 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:12.122469 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.122485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.123509 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.123513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.124763 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.124774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.125500 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.125505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.125698 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.126293 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:12.126315 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:12.128383 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:12.131479 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:12.131487 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:12.131503 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:12.131908 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:12.132284 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:12.132289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:12.132293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:12.132459 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:12.132965 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:12.132973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:12.133343 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 21:37:12.133600 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 21:37:12.134121 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":458,"threads_started":1}
I20260926 21:37:12.136428 4954 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35953
I20260926 21:37:12.136750 4954 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:12.137439 4954 heartbeater.cc:511] Master 127.0.97.62:35953 requested a full tablet report, sending...
I20260926 21:37:12.138773 4501 ts_manager.cc:194] Registered new tserver with Master: e38b2f155e1d41b19551254f787431d5 (127.0.97.3:43421)
I20260926 21:37:12.139552 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35749
I20260926 21:37:12.144958 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:37:12.146683 4689 heartbeater.cc:503] Master 127.0.97.62:35953 was elected leader, sending a full tablet report...
I20260926 21:37:12.162034 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.149789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.0.97.62:35953)
0926 21:37:12.149991 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.150001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.150444 (+ 443us) server_negotiation.cc:407] Connection header received
0926 21:37:12.150569 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.150573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.150628 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.150711 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:12.152046 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.152772 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.154001 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.154412 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.154501 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.156732 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:12.156736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:12.156740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:12.156768 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:12.159317 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.160062 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.160066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.160068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.160193 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.160560 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.160563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.160565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.160923 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:12.161110 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.161526 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.161683 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:37:12.166121 4501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:37:12.173089 4501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:37:12.207365 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.186334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41839 (local address 127.0.0.1:33210)
0926 21:37:12.199142 (+ 12808us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:12.199166 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:12.199352 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:12.200218 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:12.200229 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:12.200269 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:12.200710 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.200726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.202234 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.202239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.204802 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.204819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.206419 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.206425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.206617 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.206675 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 21:37:12.206781 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 21:37:12.206984 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12643,"thread_start_us":12493,"threads_started":1}
I20260926 21:37:12.208642 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.186895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.0.97.2:41839)
0926 21:37:12.198422 (+ 11527us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.198435 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.199517 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 21:37:12.199646 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.199657 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.199984 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.200189 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:12.200868 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.202081 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.205027 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.206171 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.207872 (+ 1701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.208079 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.208148 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.208325 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11296,"thread_start_us":11534,"threads_started":1}
I20260926 21:37:12.209939 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.186906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40421 (local address 127.0.0.1:45700)
0926 21:37:12.201651 (+ 14745us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:12.201672 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:12.201825 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:12.202668 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:12.202673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:12.202697 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:12.203232 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.203244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.205108 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.205115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.207254 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.207273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.209095 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.209100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.209286 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.209321 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:37:12.209412 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 21:37:12.209548 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14583,"mutex_wait_us":37,"thread_start_us":14455,"threads_started":1}
I20260926 21:37:12.211274 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.185997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43421 (local address 127.0.0.1:33330)
0926 21:37:12.193347 (+ 7350us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:12.193410 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:12.193623 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:12.204399 (+ 10776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:12.204403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:12.204427 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:12.204824 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.204834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.206895 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.206900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.208539 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.208552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.210453 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.210460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.210638 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.210689 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 21:37:12.210808 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 21:37:12.210932 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7017,"thread_start_us":1089,"threads_started":1}
I20260926 21:37:12.211387 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.186343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.0.97.3:43421)
0926 21:37:12.203939 (+ 17596us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.203944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.203970 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:37:12.204037 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.204042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.204213 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.204409 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:12.205018 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.206737 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.208742 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.210270 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.210408 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.210627 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.210850 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.211006 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17456,"thread_start_us":17308,"threads_started":1}
I20260926 21:37:12.212426 4757 tablet_service.cc:1467] Processing CreateTablet for tablet ac7596eb50c242bfb541c925d4d34b62 (DEFAULT_TABLE table=test-table [id=45b6855313cc465dafafaec1fb6d10d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:12.214016 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7596eb50c242bfb541c925d4d34b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:12.214812 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.187078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.0.97.1:40421)
0926 21:37:12.200246 (+ 13168us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.200259 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.201805 (+ 1546us) server_negotiation.cc:407] Connection header received
0926 21:37:12.201983 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.201991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.202247 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.202443 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:12.203438 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.204983 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.207421 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.208897 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.209030 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.209273 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.209464 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.209625 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13002,"thread_start_us":13199,"threads_started":1}
I20260926 21:37:12.215087 4889 tablet_service.cc:1467] Processing CreateTablet for tablet ac7596eb50c242bfb541c925d4d34b62 (DEFAULT_TABLE table=test-table [id=45b6855313cc465dafafaec1fb6d10d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:12.216540 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7596eb50c242bfb541c925d4d34b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:12.217949 4624 tablet_service.cc:1467] Processing CreateTablet for tablet ac7596eb50c242bfb541c925d4d34b62 (DEFAULT_TABLE table=test-table [id=45b6855313cc465dafafaec1fb6d10d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:12.219002 4624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7596eb50c242bfb541c925d4d34b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:12.230537 4972 tablet_bootstrap.cc:492] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: Bootstrap starting.
I20260926 21:37:12.233078 4973 tablet_bootstrap.cc:492] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: Bootstrap starting.
I20260926 21:37:12.233899 4974 tablet_bootstrap.cc:492] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: Bootstrap starting.
I20260926 21:37:12.235509 4972 tablet_bootstrap.cc:654] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:12.236529 4972 log.cc:824] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:12.237175 4973 tablet_bootstrap.cc:654] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:12.238472 4973 log.cc:824] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:12.239028 4974 tablet_bootstrap.cc:654] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:12.239539 4972 tablet_bootstrap.cc:492] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: No bootstrap required, opened a new log
I20260926 21:37:12.239856 4972 ts_tablet_manager.cc:1397] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 21:37:12.240276 4974 log.cc:824] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:12.241590 4973 tablet_bootstrap.cc:492] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: No bootstrap required, opened a new log
I20260926 21:37:12.241896 4973 ts_tablet_manager.cc:1397] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260926 21:37:12.243465 4974 tablet_bootstrap.cc:492] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: No bootstrap required, opened a new log
I20260926 21:37:12.243741 4974 ts_tablet_manager.cc:1397] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 21:37:12.248551 4972 raft_consensus.cc:348] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } }
I20260926 21:37:12.248960 4972 raft_consensus.cc:374] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:12.249086 4972 raft_consensus.cc:729] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3d34e544354e60b252ef0a002937b1, State: Initialized, Role: FOLLOWER
I20260926 21:37:12.249742 4972 consensus_queue.cc:260] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } }
I20260926 21:37:12.250855 4822 heartbeater.cc:503] Master 127.0.97.62:35953 was elected leader, sending a full tablet report...
I20260926 21:37:12.252679 4973 raft_consensus.cc:348] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } }
I20260926 21:37:12.253257 4973 raft_consensus.cc:374] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:12.253404 4973 raft_consensus.cc:729] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edefeaee572842b89b2b67b1adc46248, State: Initialized, Role: FOLLOWER
I20260926 21:37:12.254194 4973 consensus_queue.cc:260] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } }
I20260926 21:37:12.254794 4974 raft_consensus.cc:348] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } }
I20260926 21:37:12.255347 4974 raft_consensus.cc:374] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:12.255462 4972 ts_tablet_manager.cc:1428] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 21:37:12.255541 4974 raft_consensus.cc:729] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38b2f155e1d41b19551254f787431d5, State: Initialized, Role: FOLLOWER
I20260926 21:37:12.256369 4974 consensus_queue.cc:260] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } }
I20260926 21:37:12.257266 4973 ts_tablet_manager.cc:1428] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260926 21:37:12.257567 4954 heartbeater.cc:503] Master 127.0.97.62:35953 was elected leader, sending a full tablet report...
I20260926 21:37:12.259748 4974 ts_tablet_manager.cc:1428] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260926 21:37:12.319561 4978 raft_consensus.cc:482] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:12.319909 4978 raft_consensus.cc:504] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } }
I20260926 21:37:12.321696 4978 leader_election.cc:288] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e38b2f155e1d41b19551254f787431d5 (127.0.97.3:43421), edefeaee572842b89b2b67b1adc46248 (127.0.97.1:40421)
I20260926 21:37:12.328789 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.322147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57547 (local address 127.0.97.3:43421)
0926 21:37:12.322361 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.322368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.322904 (+ 536us) server_negotiation.cc:407] Connection header received
0926 21:37:12.323097 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.323105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.323185 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.323259 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:12.323871 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.324620 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.326784 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.327806 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.327901 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.327972 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.328371 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.328509 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 21:37:12.328879 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.322380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47571 (local address 127.0.97.1:40421)
0926 21:37:12.322580 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.322588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.323000 (+ 412us) server_negotiation.cc:407] Connection header received
0926 21:37:12.323171 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.323175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.323231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.323350 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:12.323978 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.324699 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.326973 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.327977 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.328114 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.328211 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.328518 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.328650 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:37:12.328879 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.321956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43421 (local address 127.0.97.2:57547)
0926 21:37:12.322772 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:12.322795 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:12.322926 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:12.323301 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:12.323308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:12.323336 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:12.323724 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.323734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.324749 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.324756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.326628 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.326642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.328010 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.328017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.328194 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.328229 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:37:12.328324 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 21:37:12.328414 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":98,"thread_start_us":420,"threads_started":1}
I20260926 21:37:12.328994 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.322046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40421 (local address 127.0.97.2:47571)
0926 21:37:12.322867 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:12.322886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:12.323042 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:12.323393 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:12.323397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:12.323437 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:12.323827 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.323839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.324831 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.324835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.326758 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:12.326773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.328188 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.328193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:12.328357 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.328390 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:37:12.328476 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 21:37:12.328587 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":39,"thread_start_us":498,"threads_started":1}
I20260926 21:37:12.329962 4644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7596eb50c242bfb541c925d4d34b62" candidate_uuid: "ff3d34e544354e60b252ef0a002937b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edefeaee572842b89b2b67b1adc46248" is_pre_election: true
I20260926 21:37:12.330111 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7596eb50c242bfb541c925d4d34b62" candidate_uuid: "ff3d34e544354e60b252ef0a002937b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38b2f155e1d41b19551254f787431d5" is_pre_election: true
I20260926 21:37:12.330509 4644 raft_consensus.cc:2454] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff3d34e544354e60b252ef0a002937b1 in term 0.
I20260926 21:37:12.330616 4909 raft_consensus.cc:2454] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff3d34e544354e60b252ef0a002937b1 in term 0.
I20260926 21:37:12.331197 4710 leader_election.cc:302] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edefeaee572842b89b2b67b1adc46248, ff3d34e544354e60b252ef0a002937b1; no voters:
I20260926 21:37:12.331631 4978 raft_consensus.cc:2781] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:37:12.331802 4978 raft_consensus.cc:482] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:37:12.331905 4978 raft_consensus.cc:3037] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:12.334097 4978 raft_consensus.cc:504] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } }
I20260926 21:37:12.334658 4978 leader_election.cc:288] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [CANDIDATE]: Term 1 election: Requested vote from peers e38b2f155e1d41b19551254f787431d5 (127.0.97.3:43421), edefeaee572842b89b2b67b1adc46248 (127.0.97.1:40421)
I20260926 21:37:12.335108 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7596eb50c242bfb541c925d4d34b62" candidate_uuid: "ff3d34e544354e60b252ef0a002937b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38b2f155e1d41b19551254f787431d5"
I20260926 21:37:12.335114 4644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7596eb50c242bfb541c925d4d34b62" candidate_uuid: "ff3d34e544354e60b252ef0a002937b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edefeaee572842b89b2b67b1adc46248"
I20260926 21:37:12.335328 4909 raft_consensus.cc:3037] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:12.335356 4644 raft_consensus.cc:3037] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:12.337455 4644 raft_consensus.cc:2454] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff3d34e544354e60b252ef0a002937b1 in term 1.
I20260926 21:37:12.337463 4909 raft_consensus.cc:2454] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff3d34e544354e60b252ef0a002937b1 in term 1.
I20260926 21:37:12.337833 4710 leader_election.cc:302] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edefeaee572842b89b2b67b1adc46248, ff3d34e544354e60b252ef0a002937b1; no voters:
I20260926 21:37:12.338119 4978 raft_consensus.cc:2781] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:12.338518 4978 raft_consensus.cc:686] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 1 LEADER]: Becoming Leader. State: Replica: ff3d34e544354e60b252ef0a002937b1, State: Running, Role: LEADER
I20260926 21:37:12.338982 4978 consensus_queue.cc:237] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } }
I20260926 21:37:12.343171 4501 catalog_manager.cc:5495] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff3d34e544354e60b252ef0a002937b1 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ff3d34e544354e60b252ef0a002937b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff3d34e544354e60b252ef0a002937b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 } health_report { overall_health: UNKNOWN } } }
W20260926 21:37:12.365792 4690 tablet.cc:2369] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:37:12.366125 4955 tablet.cc:2369] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:37:12.369442 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.363927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.0.97.2:41839)
0926 21:37:12.364134 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.364139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.364156 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:37:12.364368 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.364371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.364447 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.364570 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:37:12.364896 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.365764 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.367424 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.367800 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.367913 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.368330 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:37:12.368828 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:37:12.368951 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.369040 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.369116 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 21:37:12.370713 4823 tablet.cc:2369] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:37:12.377295 4644 raft_consensus.cc:1261] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [term 1 FOLLOWER]: Refusing update from remote peer ff3d34e544354e60b252ef0a002937b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:12.377317 4909 raft_consensus.cc:1261] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [term 1 FOLLOWER]: Refusing update from remote peer ff3d34e544354e60b252ef0a002937b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:12.378216 4978 consensus_queue.cc:1035] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edefeaee572842b89b2b67b1adc46248" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:12.378494 4983 consensus_queue.cc:1035] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38b2f155e1d41b19551254f787431d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:12.391772 4986 mvcc.cc:205] Tried to move back new op lower bound from 7333718558208053248 to 7333718558064099328. Current Snapshot: MvccSnapshot[applied={T|T < 7333718558208053248}]
I20260926 21:37:12.405718 4501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:12.405983 4501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:12.409312 4501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff66139901bf44dd989745bdb31b661f: Sending DeleteTablet for 3 replicas of tablet ac7596eb50c242bfb541c925d4d34b62
I20260926 21:37:12.410466 4624 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7596eb50c242bfb541c925d4d34b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45700
I20260926 21:37:12.410470 4757 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7596eb50c242bfb541c925d4d34b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33210
I20260926 21:37:12.410476 4889 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7596eb50c242bfb541c925d4d34b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33330
I20260926 21:37:12.413048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 4562
I20260926 21:37:12.422760 4997 tablet_replica.cc:330] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: stopping tablet replica
I20260926 21:37:12.423481 4997 raft_consensus.cc:2229] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:12.424068 4997 raft_consensus.cc:2258] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:12.425112 4996 tablet_replica.cc:330] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: stopping tablet replica
I20260926 21:37:12.426604 4997 ts_tablet_manager.cc:1905] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:12.433696 4995 tablet_replica.cc:330] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: stopping tablet replica
I20260926 21:37:12.433974 4996 raft_consensus.cc:2229] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:37:12.434219 4995 raft_consensus.cc:2229] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:12.434700 4996 raft_consensus.cc:2258] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:12.434778 4995 raft_consensus.cc:2258] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:12.435561 4997 ts_tablet_manager.cc:1918] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:12.435776 4997 log.cc:1196] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/ac7596eb50c242bfb541c925d4d34b62
I20260926 21:37:12.436393 4997 ts_tablet_manager.cc:1939] T ac7596eb50c242bfb541c925d4d34b62 P edefeaee572842b89b2b67b1adc46248: Deleting consensus metadata
I20260926 21:37:12.438009 4996 ts_tablet_manager.cc:1905] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:12.438346 4485 catalog_manager.cc:4841] TS edefeaee572842b89b2b67b1adc46248 (127.0.97.1:40421): tablet ac7596eb50c242bfb541c925d4d34b62 (table test-table [id=45b6855313cc465dafafaec1fb6d10d9]) successfully deleted
I20260926 21:37:12.439411 4995 ts_tablet_manager.cc:1905] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 21:37:12 dist-test-slave-lhhq krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:37:12.445842 4996 ts_tablet_manager.cc:1918] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:12.446098 4996 log.cc:1196] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/ac7596eb50c242bfb541c925d4d34b62
I20260926 21:37:12.446758 4996 ts_tablet_manager.cc:1939] T ac7596eb50c242bfb541c925d4d34b62 P ff3d34e544354e60b252ef0a002937b1: Deleting consensus metadata
I20260926 21:37:12.447067 4995 ts_tablet_manager.cc:1918] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:12.447286 4995 log.cc:1196] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/ac7596eb50c242bfb541c925d4d34b62
I20260926 21:37:12.447853 4995 ts_tablet_manager.cc:1939] T ac7596eb50c242bfb541c925d4d34b62 P e38b2f155e1d41b19551254f787431d5: Deleting consensus metadata
I20260926 21:37:12.448693 4487 catalog_manager.cc:4841] TS ff3d34e544354e60b252ef0a002937b1 (127.0.97.2:41839): tablet ac7596eb50c242bfb541c925d4d34b62 (table test-table [id=45b6855313cc465dafafaec1fb6d10d9]) successfully deleted
I20260926 21:37:12.449610 4486 catalog_manager.cc:4841] TS e38b2f155e1d41b19551254f787431d5 (127.0.97.3:43421): tablet ac7596eb50c242bfb541c925d4d34b62 (table test-table [id=45b6855313cc465dafafaec1fb6d10d9]) successfully deleted
I20260926 21:37:12.450052 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.414807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.0.97.1:40421)
0926 21:37:12.415033 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.415041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.429375 (+ 14334us) server_negotiation.cc:407] Connection header received
0926 21:37:12.429960 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.429970 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.430077 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.430163 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:12.432376 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.433748 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.435525 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.436251 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.436362 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.442398 (+ 6036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:12.442413 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:12.442431 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:12.442479 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:12.446389 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.447327 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.447336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.447340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.447492 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.447998 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.448003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.448006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.448459 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:12.448729 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.449323 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.449539 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:37:12.450757 4684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:12.547310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 4562
I20260926 21:37:12.565693 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 4695
Sep 26 21:37:12 dist-test-slave-lhhq krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:37:12.582526 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.566455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.0.97.2:41839)
0926 21:37:12.566638 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.566643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.566678 (+ 35us) server_negotiation.cc:407] Connection header received
0926 21:37:12.566776 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.566782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.566873 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.566983 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:12.568202 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.569476 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.570669 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.571347 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.571481 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.575648 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:12.575664 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:12.575693 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:12.575745 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:12.579643 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.580307 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.580317 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.580321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.580487 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.580813 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.580823 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.580827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.581312 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:12.581745 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.581900 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.582126 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:37:12.583335 4817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:12.695817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 4695
I20260926 21:37:12.714243 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 4827
Sep 26 21:37:12 dist-test-slave-lhhq krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:37:12.728730 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:12.715049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.0.97.3:43421)
0926 21:37:12.715213 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:12.715218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:12.715236 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:37:12.715358 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:12.715363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:12.715427 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:12.715516 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:12.716850 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.717665 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.719148 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:12.719544 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:12.719641 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:12.723843 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:12.723858 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:12.723874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:12.723915 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:12.726477 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.727073 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.727079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.727082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.727209 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:12.727525 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:12.727528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:12.727530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:12.727867 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:12.728063 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:12.728324 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 21:37:12.728480 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:37:12.729431 4949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:12.817014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 4827
I20260926 21:37:12.833069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 4470
I20260926 21:37:12.834201 4531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:12.939409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 4470
2026-09-26T21:37:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4117 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T21:37:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:12Z Disabled control of system clock
I20260926 21:37:13.005517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40003
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 21:37:13.326135 5006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:13.326445 5006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:13.326530 5006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:13.333878 5006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:13.333997 5006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:13.334033 5006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:13.334082 5006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:13.334126 5006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:13.343592 5006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:13.344895 5006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:13.346692 5006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:13.359025 5011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:13.359045 5012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:13.359027 5014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:13.359737 5006 server_base.cc:1034] running on GCE node
I20260926 21:37:13.360661 5006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:13.362246 5006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:13.363523 5006 hybrid_clock.cc:648] HybridClock initialized: now 1790458633363465 us; error 52 us; skew 500 ppm
I20260926 21:37:13.363938 5006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:37:13.366511 5006 webserver.cc:460] Webserver started at http://127.0.97.62:38913/ using document root <none> and password file <none>
I20260926 21:37:13.367238 5006 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:13.367376 5006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:13.367717 5006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:13.370757 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7"
format_stamp: "Formatted at 2026-09-26 21:37:13 on dist-test-slave-lhhq"
server_key: "1438ce8b3f53fc6f11caa534d0d667f3"
server_key_iv: "5d5dc17f08b7ba4a4fcef062206e1161"
server_key_version: "encryptionkey@0"
I20260926 21:37:13.371580 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7"
format_stamp: "Formatted at 2026-09-26 21:37:13 on dist-test-slave-lhhq"
server_key: "1438ce8b3f53fc6f11caa534d0d667f3"
server_key_iv: "5d5dc17f08b7ba4a4fcef062206e1161"
server_key_version: "encryptionkey@0"
I20260926 21:37:13.376901 5006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:37:13.380674 5020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:13.382138 5006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:13.382339 5006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7"
format_stamp: "Formatted at 2026-09-26 21:37:13 on dist-test-slave-lhhq"
server_key: "1438ce8b3f53fc6f11caa534d0d667f3"
server_key_iv: "5d5dc17f08b7ba4a4fcef062206e1161"
server_key_version: "encryptionkey@0"
I20260926 21:37:13.382565 5006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:13.388466 5006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:13.389308 5006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:13.389616 5006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:13.407338 5006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40003
I20260926 21:37:13.407356 5071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40003 every 8 connection(s)
I20260926 21:37:13.408797 5006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:13.412801 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:13.414309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 5006
I20260926 21:37:13.414523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:13.415155 388 external_mini_cluster.cc:1428] Setting key 3e12e4a11579d6453be08f1efafc4dd9
I20260926 21:37:13.424510 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:13.417319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.0.97.62:40003)
0926 21:37:13.418471 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:13.418483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:13.418532 (+ 49us) server_negotiation.cc:407] Connection header received
0926 21:37:13.420205 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:13.420228 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:13.420664 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:13.421077 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:13.422621 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:13.422627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:13.422644 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:13.422659 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:13.422725 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:13.422744 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:13.423041 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:13.423210 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 21:37:13.423553 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":429,"threads_started":1}
I20260926 21:37:13.425017 5072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7: Bootstrap starting.
I20260926 21:37:13.428751 5072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:13.430135 5072 log.cc:824] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:13.432898 5072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7: No bootstrap required, opened a new log
I20260926 21:37:13.440434 5072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40003 } }
I20260926 21:37:13.440892 5072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:13.441004 5072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2dd3b6c6ec48cc8149bd4b83dae2c7, State: Initialized, Role: FOLLOWER
I20260926 21:37:13.441675 5072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40003 } }
I20260926 21:37:13.441953 5072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:13.442116 5072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:13.442329 5072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:13.444576 5072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40003 } }
I20260926 21:37:13.445081 5072 leader_election.cc:302] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e2dd3b6c6ec48cc8149bd4b83dae2c7; no voters:
I20260926 21:37:13.445710 5072 leader_election.cc:288] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:13.445847 5077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:13.446535 5077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [term 1 LEADER]: Becoming Leader. State: Replica: 3e2dd3b6c6ec48cc8149bd4b83dae2c7, State: Running, Role: LEADER
I20260926 21:37:13.447117 5077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40003 } }
I20260926 21:37:13.447851 5072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:13.449970 5078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40003 } } }
I20260926 21:37:13.450017 5079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e2dd3b6c6ec48cc8149bd4b83dae2c7. Latest consensus state: current_term: 1 leader_uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dd3b6c6ec48cc8149bd4b83dae2c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40003 } } }
I20260926 21:37:13.450384 5078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:13.450382 5079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:13.451351 5083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:13.458137 5083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:13.467326 5083 catalog_manager.cc:1347] Generated new cluster ID: 219b09a5fac14245b50c5764036753e3
I20260926 21:37:13.467451 5083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:13.490975 5083 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:13.491423 5083 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:13.500550 5083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7: Generated new TSK 0
I20260926 21:37:13.501479 5083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:37:13.509030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40003
--builtin_ntp_servers=127.0.97.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 21:37:13.840159 5096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:13.840456 5096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:13.840596 5096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:13.848189 5096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:13.848364 5096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:13.848558 5096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:13.858494 5096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:13.859951 5096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:13.861785 5096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:13.874928 5101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:13.874995 5104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:13.875212 5102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:13.876451 5096 server_base.cc:1034] running on GCE node
I20260926 21:37:13.877156 5096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:13.878165 5096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:13.879401 5096 hybrid_clock.cc:648] HybridClock initialized: now 1790458633879345 us; error 52 us; skew 500 ppm
I20260926 21:37:13.879725 5096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:37:13.882053 5096 webserver.cc:460] Webserver started at http://127.0.97.1:40911/ using document root <none> and password file <none>
I20260926 21:37:13.882750 5096 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:13.882877 5096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:13.883188 5096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:13.886116 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "5d49fb423be342da93e02a5becf708b6"
format_stamp: "Formatted at 2026-09-26 21:37:13 on dist-test-slave-lhhq"
server_key: "e2505b09722dc54c0ba0e524f9c39274"
server_key_iv: "f2b9727ac53e1e187c7e581668c041e7"
server_key_version: "encryptionkey@0"
I20260926 21:37:13.886933 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5d49fb423be342da93e02a5becf708b6"
format_stamp: "Formatted at 2026-09-26 21:37:13 on dist-test-slave-lhhq"
server_key: "e2505b09722dc54c0ba0e524f9c39274"
server_key_iv: "f2b9727ac53e1e187c7e581668c041e7"
server_key_version: "encryptionkey@0"
I20260926 21:37:13.892055 5096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:37:13.895746 5110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:13.897182 5096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:37:13.897356 5096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "5d49fb423be342da93e02a5becf708b6"
format_stamp: "Formatted at 2026-09-26 21:37:13 on dist-test-slave-lhhq"
server_key: "e2505b09722dc54c0ba0e524f9c39274"
server_key_iv: "f2b9727ac53e1e187c7e581668c041e7"
server_key_version: "encryptionkey@0"
I20260926 21:37:13.897568 5096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:13.903246 5096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:13.904037 5096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:13.904282 5096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:13.905218 5096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:13.906665 5096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:13.906749 5096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:13.906826 5096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:13.906883 5096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:13.939901 5096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33859
I20260926 21:37:13.939990 5222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33859 every 8 connection(s)
I20260926 21:37:13.941356 5096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:13.950567 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:13.944329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57309 (local address 127.0.97.62:40003)
0926 21:37:13.944779 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:13.944783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:13.946296 (+ 1513us) server_negotiation.cc:407] Connection header received
0926 21:37:13.948055 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:13.948060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:13.948113 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:13.948219 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:13.949917 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:13.949921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:13.949925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:13.949942 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:13.949983 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:13.949996 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:13.950100 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:13.950274 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 21:37:13.950325 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
I20260926 21:37:13.951238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 5096
I20260926 21:37:13.951448 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:37:13.951753 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:13.944978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40003 (local address 127.0.97.1:57309)
0926 21:37:13.946119 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:13.946160 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:13.947718 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:13.948646 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:13.948656 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:13.948767 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:13.949586 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:37:13.949588 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:13.949654 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:37:13.949746 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:13.950084 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:13.950091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:13.950097 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:37:13.950377 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 21:37:13.950707 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":341,"threads_started":1}
I20260926 21:37:13.952121 388 external_mini_cluster.cc:1428] Setting key c87a71235807ef66218acf0ed3e9b85e
I20260926 21:37:13.953547 5223 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40003
I20260926 21:37:13.953891 5223 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:13.954486 5223 heartbeater.cc:511] Master 127.0.97.62:40003 requested a full tablet report, sending...
I20260926 21:37:13.955161 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40003
--builtin_ntp_servers=127.0.97.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 21:37:13.956781 5037 ts_manager.cc:194] Registered new tserver with Master: 5d49fb423be342da93e02a5becf708b6 (127.0.97.1:33859)
W20260926 21:37:14.285873 5227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:14.286228 5227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:14.286378 5227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:14.293701 5227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:14.293836 5227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:14.294019 5227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:14.303287 5227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:14.304597 5227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:14.306408 5227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:14.319495 5232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:14.319664 5235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:14.319653 5233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:14.320502 5227 server_base.cc:1034] running on GCE node
I20260926 21:37:14.321115 5227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:14.322135 5227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:14.323386 5227 hybrid_clock.cc:648] HybridClock initialized: now 1790458634323312 us; error 51 us; skew 500 ppm
I20260926 21:37:14.323721 5227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:37:14.326084 5227 webserver.cc:460] Webserver started at http://127.0.97.2:39945/ using document root <none> and password file <none>
I20260926 21:37:14.326776 5227 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:14.326915 5227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:14.327230 5227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:14.330155 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "0a3c390447a64f7b92fa252f3d4c5891"
format_stamp: "Formatted at 2026-09-26 21:37:14 on dist-test-slave-lhhq"
server_key: "56bf6e1288fe62316f586459dd3270fa"
server_key_iv: "96f69abc5b4807a596a68b5b6e7af948"
server_key_version: "encryptionkey@0"
I20260926 21:37:14.330943 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0a3c390447a64f7b92fa252f3d4c5891"
format_stamp: "Formatted at 2026-09-26 21:37:14 on dist-test-slave-lhhq"
server_key: "56bf6e1288fe62316f586459dd3270fa"
server_key_iv: "96f69abc5b4807a596a68b5b6e7af948"
server_key_version: "encryptionkey@0"
I20260926 21:37:14.336297 5227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:37:14.340000 5241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:14.341598 5227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:37:14.341784 5227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "0a3c390447a64f7b92fa252f3d4c5891"
format_stamp: "Formatted at 2026-09-26 21:37:14 on dist-test-slave-lhhq"
server_key: "56bf6e1288fe62316f586459dd3270fa"
server_key_iv: "96f69abc5b4807a596a68b5b6e7af948"
server_key_version: "encryptionkey@0"
I20260926 21:37:14.342041 5227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:14.347968 5227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:14.348851 5227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:14.349129 5227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:14.350135 5227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:14.351608 5227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:14.351730 5227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:14.351866 5227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:14.351958 5227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:14.384433 5227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36239
I20260926 21:37:14.384476 5353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36239 every 8 connection(s)
I20260926 21:37:14.385944 5227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:14.394488 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.388886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33395 (local address 127.0.97.62:40003)
0926 21:37:14.389068 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:14.389073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:14.390469 (+ 1396us) server_negotiation.cc:407] Connection header received
0926 21:37:14.392186 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:14.392190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:14.392260 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:14.392366 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:14.393885 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:14.393890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:14.393893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:14.393909 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:14.393941 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:14.393953 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:14.394052 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:14.394238 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 21:37:14.394288 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:37:14.395511 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.389399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40003 (local address 127.0.97.2:33395)
0926 21:37:14.390309 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:14.390353 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:14.391905 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:14.392576 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:14.392584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:14.392742 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:14.393579 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:37:14.393582 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:14.393620 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:37:14.393706 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:14.394046 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:14.394053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:14.394058 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:37:14.394276 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 21:37:14.394478 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":281,"threads_started":1}
I20260926 21:37:14.396167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 5227
I20260926 21:37:14.396378 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:37:14.396944 388 external_mini_cluster.cc:1428] Setting key 7c954438a2d4481b45724e73f7185ad0
I20260926 21:37:14.397159 5354 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40003
I20260926 21:37:14.397501 5354 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:14.398113 5354 heartbeater.cc:511] Master 127.0.97.62:40003 requested a full tablet report, sending...
I20260926 21:37:14.399549 5037 ts_manager.cc:194] Registered new tserver with Master: 0a3c390447a64f7b92fa252f3d4c5891 (127.0.97.2:36239)
I20260926 21:37:14.399737 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40003
--builtin_ntp_servers=127.0.97.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 21:37:14.726015 5357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:14.726297 5357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:14.726385 5357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:14.733682 5357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:14.733789 5357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:14.733897 5357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:14.743207 5357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:14.744490 5357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:14.746289 5357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:14.759809 5363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:14.759850 5362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:14.759770 5365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:14.760725 5357 server_base.cc:1034] running on GCE node
I20260926 21:37:14.761387 5357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:14.762305 5357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:14.763500 5357 hybrid_clock.cc:648] HybridClock initialized: now 1790458634763453 us; error 34 us; skew 500 ppm
I20260926 21:37:14.763839 5357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:37:14.766180 5357 webserver.cc:460] Webserver started at http://127.0.97.3:38211/ using document root <none> and password file <none>
I20260926 21:37:14.766881 5357 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:14.766966 5357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:14.767326 5357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:14.770295 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "1a543ec7ec454a458b07441f3d1bc400"
format_stamp: "Formatted at 2026-09-26 21:37:14 on dist-test-slave-lhhq"
server_key: "554c49758750367cb3792d2a04539a09"
server_key_iv: "fb8ee49988f9b797d3061ca8ad966cce"
server_key_version: "encryptionkey@0"
I20260926 21:37:14.771108 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1a543ec7ec454a458b07441f3d1bc400"
format_stamp: "Formatted at 2026-09-26 21:37:14 on dist-test-slave-lhhq"
server_key: "554c49758750367cb3792d2a04539a09"
server_key_iv: "fb8ee49988f9b797d3061ca8ad966cce"
server_key_version: "encryptionkey@0"
I20260926 21:37:14.776223 5357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:37:14.779897 5371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:14.781365 5357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:14.781529 5357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "1a543ec7ec454a458b07441f3d1bc400"
format_stamp: "Formatted at 2026-09-26 21:37:14 on dist-test-slave-lhhq"
server_key: "554c49758750367cb3792d2a04539a09"
server_key_iv: "fb8ee49988f9b797d3061ca8ad966cce"
server_key_version: "encryptionkey@0"
I20260926 21:37:14.781787 5357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:14.787569 5357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:14.788406 5357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:14.788684 5357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:14.789666 5357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:14.791070 5357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:14.791194 5357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:14.791339 5357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:14.791429 5357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:14.824224 5357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37717
I20260926 21:37:14.824270 5483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37717 every 8 connection(s)
I20260926 21:37:14.825616 5357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:14.829429 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 5357
I20260926 21:37:14.829612 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:37:14.830140 388 external_mini_cluster.cc:1428] Setting key 7f66635fad7a1c56995307002e79b023
I20260926 21:37:14.834327 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.828154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45001 (local address 127.0.97.62:40003)
0926 21:37:14.828370 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:14.828375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:14.830029 (+ 1654us) server_negotiation.cc:407] Connection header received
0926 21:37:14.832140 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:14.832146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:14.832220 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:14.832337 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:14.833700 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:14.833704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:14.833709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:14.833726 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:14.833758 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:14.833769 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:14.833868 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:14.834050 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 21:37:14.834096 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 21:37:14.835244 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.828646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40003 (local address 127.0.97.3:45001)
0926 21:37:14.829843 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:14.829892 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:14.831865 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:14.832567 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:14.832576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:14.832670 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:14.833420 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:37:14.833423 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:14.833460 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:37:14.833542 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:14.833870 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:14.833877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:14.833883 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:37:14.834098 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 21:37:14.834291 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":319,"threads_started":1}
I20260926 21:37:14.836522 5484 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40003
I20260926 21:37:14.836834 5484 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:14.837455 5484 heartbeater.cc:511] Master 127.0.97.62:40003 requested a full tablet report, sending...
I20260926 21:37:14.838757 5037 ts_manager.cc:194] Registered new tserver with Master: 1a543ec7ec454a458b07441f3d1bc400 (127.0.97.3:37717)
I20260926 21:37:14.845741 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:37:14.852918 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.850567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.0.97.62:40003)
0926 21:37:14.850721 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:14.850726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:14.851209 (+ 483us) server_negotiation.cc:407] Connection header received
0926 21:37:14.851354 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:14.851358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:14.851408 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:14.851490 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:14.852382 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:14.852386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:14.852389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:14.852403 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:14.852431 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:14.852442 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:14.852538 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:14.852661 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 21:37:14.852717 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 21:37:14.856807 5037 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:37:14.863531 5037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:37:14.882789 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.875940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.0.97.1:33859)
0926 21:37:14.877083 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:14.877097 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:14.878470 (+ 1373us) server_negotiation.cc:407] Connection header received
0926 21:37:14.878703 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:14.878712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:14.878959 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:14.879090 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:14.880979 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:14.880988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:14.881004 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:14.881024 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:14.881094 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:14.881120 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:14.881275 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:14.882370 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0926 21:37:14.882481 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":866,"threads_started":1}
I20260926 21:37:14.883476 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.875392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33859 (local address 127.0.0.1:58010)
0926 21:37:14.878369 (+ 2977us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:14.878387 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:14.878549 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:14.879858 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:14.879864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:14.879936 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:14.880670 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:37:14.880673 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:14.880698 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:37:14.880829 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:14.881326 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:14.881335 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:14.881342 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 21:37:14.881510 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 21:37:14.881622 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2841,"thread_start_us":1900,"threads_started":1}
I20260926 21:37:14.884111 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.875296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36239 (local address 127.0.0.1:52710)
0926 21:37:14.877714 (+ 2418us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:14.877756 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:14.877872 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:14.881668 (+ 3796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:14.881673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:14.881693 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:14.882366 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:37:14.882369 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:14.882385 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:37:14.882424 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:14.883614 (+ 1190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:14.883622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:14.883628 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:37:14.883748 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 21:37:14.883801 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2187,"thread_start_us":2758,"threads_started":1}
I20260926 21:37:14.884740 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.879389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37717 (local address 127.0.0.1:52018)
0926 21:37:14.881004 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:14.881025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:14.881129 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:14.882745 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:14.882751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:14.882777 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:14.883401 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:37:14.883406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:14.883424 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:37:14.883468 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:14.884126 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:14.884130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:14.884135 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:37:14.884264 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 21:37:14.884308 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"thread_start_us":1316,"threads_started":1}
I20260926 21:37:14.884977 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.875481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.0.97.2:36239)
0926 21:37:14.881127 (+ 5646us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:14.881136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:14.881165 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:37:14.881271 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:14.881279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:14.881501 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:14.882112 (+ 611us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:14.883364 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:14.883373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:14.883385 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:14.883409 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:14.883468 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:14.883490 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:14.884470 (+ 980us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:14.884523 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 21:37:14.884644 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5472,"thread_start_us":4883,"threads_started":1}
I20260926 21:37:14.885463 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:14.878856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.0.97.3:37717)
0926 21:37:14.881706 (+ 2850us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:14.881713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:14.881744 (+ 31us) server_negotiation.cc:407] Connection header received
0926 21:37:14.881810 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:14.881816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:14.882021 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:14.882169 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:14.883767 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:14.883773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:14.883805 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:14.883824 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:14.883888 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:14.883907 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:14.884068 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:14.884968 (+ 900us) server_negotiation.cc:299] Negotiation successful
0926 21:37:14.885052 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2697,"thread_start_us":2522,"threads_started":1}
I20260926 21:37:14.887702 5158 tablet_service.cc:1467] Processing CreateTablet for tablet db17f80a980c4bb9a8ceeec1236ec09c (DEFAULT_TABLE table=test-table [id=87121d23965b4dd5b9e95cdf1476b3c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:14.889222 5158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db17f80a980c4bb9a8ceeec1236ec09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:14.888795 5289 tablet_service.cc:1467] Processing CreateTablet for tablet db17f80a980c4bb9a8ceeec1236ec09c (DEFAULT_TABLE table=test-table [id=87121d23965b4dd5b9e95cdf1476b3c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:14.888933 5419 tablet_service.cc:1467] Processing CreateTablet for tablet db17f80a980c4bb9a8ceeec1236ec09c (DEFAULT_TABLE table=test-table [id=87121d23965b4dd5b9e95cdf1476b3c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:14.890134 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db17f80a980c4bb9a8ceeec1236ec09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:14.890141 5289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db17f80a980c4bb9a8ceeec1236ec09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:14.901038 5502 tablet_bootstrap.cc:492] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: Bootstrap starting.
I20260926 21:37:14.903551 5503 tablet_bootstrap.cc:492] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: Bootstrap starting.
I20260926 21:37:14.904572 5504 tablet_bootstrap.cc:492] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: Bootstrap starting.
I20260926 21:37:14.906910 5503 tablet_bootstrap.cc:654] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:14.906915 5502 tablet_bootstrap.cc:654] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:14.907351 5504 tablet_bootstrap.cc:654] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:14.908170 5503 log.cc:824] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:14.908170 5502 log.cc:824] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:14.908329 5504 log.cc:824] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:14.911293 5503 tablet_bootstrap.cc:492] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: No bootstrap required, opened a new log
I20260926 21:37:14.911568 5503 ts_tablet_manager.cc:1397] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 21:37:14.911607 5504 tablet_bootstrap.cc:492] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: No bootstrap required, opened a new log
I20260926 21:37:14.911765 5502 tablet_bootstrap.cc:492] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: No bootstrap required, opened a new log
I20260926 21:37:14.911880 5504 ts_tablet_manager.cc:1397] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 21:37:14.912021 5502 ts_tablet_manager.cc:1397] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260926 21:37:14.919128 5504 raft_consensus.cc:348] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } }
I20260926 21:37:14.919539 5504 raft_consensus.cc:374] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:14.919637 5504 raft_consensus.cc:729] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d49fb423be342da93e02a5becf708b6, State: Initialized, Role: FOLLOWER
I20260926 21:37:14.920297 5504 consensus_queue.cc:260] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } }
I20260926 21:37:14.921473 5223 heartbeater.cc:503] Master 127.0.97.62:40003 was elected leader, sending a full tablet report...
I20260926 21:37:14.922019 5504 ts_tablet_manager.cc:1428] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260926 21:37:14.922251 5503 raft_consensus.cc:348] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } }
I20260926 21:37:14.922681 5503 raft_consensus.cc:374] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:14.922784 5503 raft_consensus.cc:729] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3c390447a64f7b92fa252f3d4c5891, State: Initialized, Role: FOLLOWER
I20260926 21:37:14.923138 5502 raft_consensus.cc:348] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } }
I20260926 21:37:14.923410 5503 consensus_queue.cc:260] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } }
I20260926 21:37:14.923566 5502 raft_consensus.cc:374] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:14.923656 5502 raft_consensus.cc:729] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a543ec7ec454a458b07441f3d1bc400, State: Initialized, Role: FOLLOWER
I20260926 21:37:14.924392 5354 heartbeater.cc:503] Master 127.0.97.62:40003 was elected leader, sending a full tablet report...
I20260926 21:37:14.924968 5502 consensus_queue.cc:260] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } }
I20260926 21:37:14.926182 5503 ts_tablet_manager.cc:1428] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260926 21:37:14.927184 5484 heartbeater.cc:503] Master 127.0.97.62:40003 was elected leader, sending a full tablet report...
I20260926 21:37:14.927739 5502 ts_tablet_manager.cc:1428] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260926 21:37:14.943568 5224 tablet.cc:2369] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:37:15.031927 5508 raft_consensus.cc:482] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:15.032210 5508 raft_consensus.cc:504] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } }
I20260926 21:37:15.033862 5508 leader_election.cc:288] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a3c390447a64f7b92fa252f3d4c5891 (127.0.97.2:36239), 1a543ec7ec454a458b07441f3d1bc400 (127.0.97.3:37717)
I20260926 21:37:15.037074 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.034305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53059 (local address 127.0.97.2:36239)
0926 21:37:15.034541 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:15.034551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:15.034997 (+ 446us) server_negotiation.cc:407] Connection header received
0926 21:37:15.035184 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:15.035191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:15.035261 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:15.035364 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:15.036445 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:15.036451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:15.036456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:15.036474 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:15.036513 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:15.036525 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:15.036653 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:15.036756 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 21:37:15.036814 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 21:37:15.037293 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.034261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36239 (local address 127.0.97.1:53059)
0926 21:37:15.034876 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:15.034892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:15.035043 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:15.035486 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:15.035490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:15.035508 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:15.036230 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:37:15.036232 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:15.036249 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:37:15.036289 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:15.036623 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:15.036629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:15.036635 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:37:15.036771 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 21:37:15.036811 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":360,"threads_started":1}
I20260926 21:37:15.037748 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.035139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47643 (local address 127.0.97.3:37717)
0926 21:37:15.035379 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:15.035384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:15.035714 (+ 330us) server_negotiation.cc:407] Connection header received
0926 21:37:15.035884 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:15.035890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:15.035942 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:15.036043 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:15.036987 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:15.036993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:15.036998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:15.037015 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:15.037049 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:15.037065 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:15.037221 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:15.037505 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 21:37:15.037571 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 21:37:15.037977 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.035018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37717 (local address 127.0.97.1:47643)
0926 21:37:15.035598 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:15.035620 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:15.035747 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:15.036059 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:15.036063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:15.036086 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:15.036766 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:37:15.036769 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:15.036785 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:37:15.036824 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:15.037290 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:15.037294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:15.037299 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:37:15.037530 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 21:37:15.037581 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":258,"threads_started":1}
I20260926 21:37:15.038362 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db17f80a980c4bb9a8ceeec1236ec09c" candidate_uuid: "5d49fb423be342da93e02a5becf708b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3c390447a64f7b92fa252f3d4c5891" is_pre_election: true
I20260926 21:37:15.038858 5309 raft_consensus.cc:2454] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d49fb423be342da93e02a5becf708b6 in term 0.
I20260926 21:37:15.038925 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db17f80a980c4bb9a8ceeec1236ec09c" candidate_uuid: "5d49fb423be342da93e02a5becf708b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a543ec7ec454a458b07441f3d1bc400" is_pre_election: true
I20260926 21:37:15.039424 5439 raft_consensus.cc:2454] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d49fb423be342da93e02a5becf708b6 in term 0.
I20260926 21:37:15.039542 5114 leader_election.cc:302] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3c390447a64f7b92fa252f3d4c5891, 5d49fb423be342da93e02a5becf708b6; no voters:
I20260926 21:37:15.039965 5508 raft_consensus.cc:2781] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:37:15.040081 5508 raft_consensus.cc:482] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:37:15.040231 5508 raft_consensus.cc:3037] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:15.042412 5508 raft_consensus.cc:504] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } }
I20260926 21:37:15.043082 5508 leader_election.cc:288] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [CANDIDATE]: Term 1 election: Requested vote from peers 0a3c390447a64f7b92fa252f3d4c5891 (127.0.97.2:36239), 1a543ec7ec454a458b07441f3d1bc400 (127.0.97.3:37717)
I20260926 21:37:15.043684 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db17f80a980c4bb9a8ceeec1236ec09c" candidate_uuid: "5d49fb423be342da93e02a5becf708b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3c390447a64f7b92fa252f3d4c5891"
I20260926 21:37:15.043768 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db17f80a980c4bb9a8ceeec1236ec09c" candidate_uuid: "5d49fb423be342da93e02a5becf708b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a543ec7ec454a458b07441f3d1bc400"
I20260926 21:37:15.043915 5309 raft_consensus.cc:3037] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:15.043952 5439 raft_consensus.cc:3037] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:15.047251 5309 raft_consensus.cc:2454] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d49fb423be342da93e02a5becf708b6 in term 1.
I20260926 21:37:15.047250 5439 raft_consensus.cc:2454] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d49fb423be342da93e02a5becf708b6 in term 1.
I20260926 21:37:15.047690 5114 leader_election.cc:302] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a543ec7ec454a458b07441f3d1bc400, 5d49fb423be342da93e02a5becf708b6; no voters:
I20260926 21:37:15.047966 5508 raft_consensus.cc:2781] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:15.048412 5508 raft_consensus.cc:686] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 1 LEADER]: Becoming Leader. State: Replica: 5d49fb423be342da93e02a5becf708b6, State: Running, Role: LEADER
I20260926 21:37:15.048930 5508 consensus_queue.cc:237] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } }
I20260926 21:37:15.054265 5037 catalog_manager.cc:5495] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d49fb423be342da93e02a5becf708b6 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5d49fb423be342da93e02a5becf708b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d49fb423be342da93e02a5becf708b6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 } health_report { overall_health: UNKNOWN } } }
I20260926 21:37:15.073958 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.071865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.0.97.1:33859)
0926 21:37:15.072064 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:15.072071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:15.072096 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:37:15.072181 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:15.072185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:15.072269 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:15.072440 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:15.073344 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:15.073349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:15.073356 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:15.073376 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:15.073404 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:15.073417 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:15.073530 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:15.073627 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 21:37:15.073685 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 21:37:15.077297 5485 tablet.cc:2369] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:37:15.082394 5309 raft_consensus.cc:1261] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [term 1 FOLLOWER]: Refusing update from remote peer 5d49fb423be342da93e02a5becf708b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:15.082396 5439 raft_consensus.cc:1261] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [term 1 FOLLOWER]: Refusing update from remote peer 5d49fb423be342da93e02a5becf708b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:15.083340 5508 consensus_queue.cc:1035] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a3c390447a64f7b92fa252f3d4c5891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:15.083664 5513 consensus_queue.cc:1035] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a543ec7ec454a458b07441f3d1bc400" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:15.109581 5037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:15.109865 5037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:15.113340 5037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e2dd3b6c6ec48cc8149bd4b83dae2c7: Sending DeleteTablet for 3 replicas of tablet db17f80a980c4bb9a8ceeec1236ec09c
I20260926 21:37:15.114529 5158 tablet_service.cc:1514] Processing DeleteTablet for tablet db17f80a980c4bb9a8ceeec1236ec09c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:15 UTC) from {username='slave'} at 127.0.0.1:58010
I20260926 21:37:15.114853 5289 tablet_service.cc:1514] Processing DeleteTablet for tablet db17f80a980c4bb9a8ceeec1236ec09c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:15 UTC) from {username='slave'} at 127.0.0.1:52710
I20260926 21:37:15.115568 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet db17f80a980c4bb9a8ceeec1236ec09c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:15 UTC) from {username='slave'} at 127.0.0.1:52018
I20260926 21:37:15.116770 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 5096
I20260926 21:37:15.119515 5527 tablet_replica.cc:330] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: stopping tablet replica
I20260926 21:37:15.120050 5527 raft_consensus.cc:2229] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:15.120615 5527 raft_consensus.cc:2258] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:15.121455 5526 tablet_replica.cc:330] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: stopping tablet replica
I20260926 21:37:15.121879 5526 raft_consensus.cc:2229] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:37:15.122552 5526 raft_consensus.cc:2258] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:15.124130 5527 ts_tablet_manager.cc:1905] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:15.126770 5526 ts_tablet_manager.cc:1905] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:15.129469 5528 tablet_replica.cc:330] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: stopping tablet replica
I20260926 21:37:15.129895 5528 raft_consensus.cc:2229] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:15.130375 5528 raft_consensus.cc:2258] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:15.131474 5527 ts_tablet_manager.cc:1918] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:15.131690 5527 log.cc:1196] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/db17f80a980c4bb9a8ceeec1236ec09c
I20260926 21:37:15.132239 5527 ts_tablet_manager.cc:1939] T db17f80a980c4bb9a8ceeec1236ec09c P 0a3c390447a64f7b92fa252f3d4c5891: Deleting consensus metadata
I20260926 21:37:15.132299 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.118649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.0.97.1:33859)
0926 21:37:15.118801 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:15.118809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:15.125809 (+ 7000us) server_negotiation.cc:407] Connection header received
0926 21:37:15.126137 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:15.126145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:15.126226 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:15.126330 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:15.131415 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:15.131421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:15.131426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:15.131447 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:15.131482 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:15.131498 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:15.131600 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:15.131896 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 21:37:15.131966 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 21:37:15.132639 5528 ts_tablet_manager.cc:1905] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:15.133370 5218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:15.134215 5024 catalog_manager.cc:4841] TS 0a3c390447a64f7b92fa252f3d4c5891 (127.0.97.2:36239): tablet db17f80a980c4bb9a8ceeec1236ec09c (table test-table [id=87121d23965b4dd5b9e95cdf1476b3c2]) successfully deleted
I20260926 21:37:15.139810 5528 ts_tablet_manager.cc:1918] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:15.139936 5528 log.cc:1196] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/db17f80a980c4bb9a8ceeec1236ec09c
I20260926 21:37:15.140389 5528 ts_tablet_manager.cc:1939] T db17f80a980c4bb9a8ceeec1236ec09c P 1a543ec7ec454a458b07441f3d1bc400: Deleting consensus metadata
I20260926 21:37:15.141918 5024 catalog_manager.cc:4841] TS 1a543ec7ec454a458b07441f3d1bc400 (127.0.97.3:37717): tablet db17f80a980c4bb9a8ceeec1236ec09c (table test-table [id=87121d23965b4dd5b9e95cdf1476b3c2]) successfully deleted
I20260926 21:37:15.230312 5526 ts_tablet_manager.cc:1918] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:15.230597 5526 log.cc:1196] T db17f80a980c4bb9a8ceeec1236ec09c P 5d49fb423be342da93e02a5becf708b6: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/db17f80a980c4bb9a8ceeec1236ec09c
I20260926 21:37:15.231026 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 5096
I20260926 21:37:15.249420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 5227
W20260926 21:37:15.249296 5023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33859 (error 108)
I20260926 21:37:15.251130 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.250705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33859 (local address 127.0.0.1:58038)
0926 21:37:15.250879 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:15.250998 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:37:15.251368 5023 catalog_manager.cc:4584] TS 5d49fb423be342da93e02a5becf708b6 (127.0.97.1:33859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db17f80a980c4bb9a8ceeec1236ec09c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33859: connect: Connection refused (error 111)
I20260926 21:37:15.252696 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.250332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.0.97.2:36239)
0926 21:37:15.250494 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:15.250501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:15.250526 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:37:15.250757 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:15.250762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:15.250842 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:15.250957 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:15.252029 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:15.252035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:15.252046 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:15.252061 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:15.252089 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:15.252101 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:15.252233 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:15.252321 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 21:37:15.252377 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:37:15.253569 5349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:15.305069 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.304561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33859 (local address 127.0.0.1:58054)
0926 21:37:15.304808 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:15.304913 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 21:37:15.306146 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.305800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33859 (local address 127.0.0.1:58068)
0926 21:37:15.305969 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:15.306034 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 21:37:15.341140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 5227
I20260926 21:37:15.357723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 5357
I20260926 21:37:15.360405 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.358558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.0.97.3:37717)
0926 21:37:15.358752 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:15.358756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:15.358775 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:37:15.358859 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:15.358862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:15.358934 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:15.359017 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:15.359809 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:15.359813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:15.359816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:15.359832 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:15.359862 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:15.359873 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:15.359975 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:15.360034 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 21:37:15.360077 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:37:15.360994 5479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:15.366359 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.365915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33859 (local address 127.0.0.1:58076)
0926 21:37:15.366123 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:15.366217 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 21:37:15.367267 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.366939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33859 (local address 127.0.0.1:58086)
0926 21:37:15.367080 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:15.367162 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 21:37:15.447916 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.447408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33859 (local address 127.0.0.1:58102)
0926 21:37:15.447638 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:15.447742 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 21:37:15.449100 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:15.448621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33859 (local address 127.0.0.1:58110)
0926 21:37:15.448868 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:15.448935 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 21:37:15.462560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 5357
I20260926 21:37:15.478780 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 5006
I20260926 21:37:15.479941 5067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:15.583057 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 5006
2026-09-26T21:37:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2643 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 21:37:15.623939 388 mini_oidc.cc:152] Starting JWKS server
I20260926 21:37:15.626565 388 webserver.cc:460] Webserver started at https://127.0.0.1:41307/ using document root <none> and password file <none>
I20260926 21:37:15.626878 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 21:37:15.627506 388 webserver.cc:460] Webserver started at http://127.0.0.1:41897/ using document root <none> and password file <none>
2026-09-26T21:37:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:15Z Disabled control of system clock
I20260926 21:37:15.653179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33749
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33749
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41307/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/testchainca.cert with env {}
W20260926 21:37:15.980681 5540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:15.981029 5540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:15.981240 5540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:15.988622 5540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:15.988785 5540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:15.988878 5540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:15.988983 5540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:15.989066 5540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 21:37:15.989153 5540 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 21:37:15.989257 5540 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41307/jwks.json
I20260926 21:37:15.999042 5540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41307/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:16.000402 5540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:16.001797 5540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:16.013897 5546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:16.013957 5548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:16.013983 5545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:16.015204 5540 server_base.cc:1034] running on GCE node
I20260926 21:37:16.016016 5540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:16.017467 5540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:16.018690 5540 hybrid_clock.cc:648] HybridClock initialized: now 1790458636018637 us; error 40 us; skew 500 ppm
I20260926 21:37:16.019045 5540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:37:16.021445 5540 webserver.cc:460] Webserver started at http://127.0.97.62:44241/ using document root <none> and password file <none>
I20260926 21:37:16.022112 5540 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:16.022202 5540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:16.022449 5540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:16.025307 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "a9a1bd0a37bd478c98a375d151985773"
format_stamp: "Formatted at 2026-09-26 21:37:16 on dist-test-slave-lhhq"
server_key: "db20fb72cd01893fe139abd783545585"
server_key_iv: "fe6bc1e31f94be273e12507ad3d9b79b"
server_key_version: "encryptionkey@0"
I20260926 21:37:16.026043 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "a9a1bd0a37bd478c98a375d151985773"
format_stamp: "Formatted at 2026-09-26 21:37:16 on dist-test-slave-lhhq"
server_key: "db20fb72cd01893fe139abd783545585"
server_key_iv: "fe6bc1e31f94be273e12507ad3d9b79b"
server_key_version: "encryptionkey@0"
I20260926 21:37:16.031013 5540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 21:37:16.034268 5554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:16.035567 5540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:37:16.035745 5540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "a9a1bd0a37bd478c98a375d151985773"
format_stamp: "Formatted at 2026-09-26 21:37:16 on dist-test-slave-lhhq"
server_key: "db20fb72cd01893fe139abd783545585"
server_key_iv: "fe6bc1e31f94be273e12507ad3d9b79b"
server_key_version: "encryptionkey@0"
I20260926 21:37:16.035976 5540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:16.051290 5540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:16.052146 5540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:16.052453 5540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:16.070571 5540 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33749
I20260926 21:37:16.070591 5605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33749 every 8 connection(s)
I20260926 21:37:16.072077 5540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:16.075908 5606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:16.081991 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 5540
I20260926 21:37:16.082175 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:16.082801 388 external_mini_cluster.cc:1428] Setting key f10ad158e72ba315cb1381fda97e7faf
I20260926 21:37:16.086678 5606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773: Bootstrap starting.
I20260926 21:37:16.090036 5606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:16.091259 5606 log.cc:824] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:16.093935 5606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773: No bootstrap required, opened a new log
I20260926 21:37:16.096608 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:16.084926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.0.97.62:33749)
0926 21:37:16.085922 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:16.085933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:16.085973 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:37:16.087652 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:16.087673 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:16.088147 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:16.088470 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:16.089091 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.090445 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.092255 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.092907 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.093105 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:16.094419 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:16.094426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:16.094442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:16.094950 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:16.095021 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:16.095040 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:16.095168 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:16.095386 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 21:37:16.095670 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":371,"threads_started":1}
I20260926 21:37:16.100307 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:37:16.101781 5606 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a1bd0a37bd478c98a375d151985773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33749 } }
I20260926 21:37:16.102138 5606 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:16.102241 5606 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a1bd0a37bd478c98a375d151985773, State: Initialized, Role: FOLLOWER
I20260926 21:37:16.102828 5606 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a1bd0a37bd478c98a375d151985773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33749 } }
I20260926 21:37:16.103066 5606 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:16.103204 5606 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:16.103407 5606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:16.105616 5606 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a1bd0a37bd478c98a375d151985773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33749 } }
I20260926 21:37:16.106092 5606 leader_election.cc:302] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a1bd0a37bd478c98a375d151985773; no voters:
I20260926 21:37:16.106642 5606 leader_election.cc:288] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:16.106796 5611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:16.107513 5611 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [term 1 LEADER]: Becoming Leader. State: Replica: a9a1bd0a37bd478c98a375d151985773, State: Running, Role: LEADER
I20260926 21:37:16.108008 5611 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a1bd0a37bd478c98a375d151985773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33749 } }
I20260926 21:37:16.108754 5606 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:16.111537 5612 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a1bd0a37bd478c98a375d151985773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a1bd0a37bd478c98a375d151985773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33749 } } }
I20260926 21:37:16.111891 5612 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:16.111603 5613 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a1bd0a37bd478c98a375d151985773. Latest consensus state: current_term: 1 leader_uuid: "a9a1bd0a37bd478c98a375d151985773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a1bd0a37bd478c98a375d151985773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33749 } } }
I20260926 21:37:16.112025 5613 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:16.113960 5617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:16.119285 5617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:16.127758 5617 catalog_manager.cc:1347] Generated new cluster ID: 48c7aa0bb4754c79854d420d15843cb4
I20260926 21:37:16.127866 5617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:16.154616 5617 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:16.155943 5617 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:16.170377 5617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9a1bd0a37bd478c98a375d151985773: Generated new TSK 0
I20260926 21:37:16.171509 5617 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 21:37:16.217981 5636 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 21:37:16.218163 5636 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 21:37:16.219059 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:16.197299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.0.97.62:33749)
0926 21:37:16.197539 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:16.197550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:16.198027 (+ 477us) server_negotiation.cc:407] Connection header received
0926 21:37:16.198397 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:16.198401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:16.198471 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:16.198589 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0926 21:37:16.199187 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.199937 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.202091 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.202559 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.202652 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:16.203260 (+ 608us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 21:37:16.218565 (+ 15305us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 21:37:16.218862 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 21:37:16.219198 5609 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 21:37:16.221091 5635 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:16.197167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33749 (local address 127.0.0.1:44286)
0926 21:37:16.197916 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:16.197937 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:16.198123 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:16.198649 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:16.198654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:16.198667 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 21:37:16.199037 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:16.199048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.200084 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.200090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:16.201953 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:16.201968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.202678 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.202726 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:16.202920 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:16.203001 (+ 81us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 21:37:16.218886 (+ 15885us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 21:37:16.219279 (+ 393us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33749: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":354,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:33749: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 21:37:16.222952 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 5540
I20260926 21:37:16.224118 5601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:16.318305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 5540
2026-09-26T21:37:16Z chronyd exiting
I20260926 21:37:16.368863 388 test_util.cc:175] -----------------------------------------------
I20260926 21:37:16.369021 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790458593267894-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (749 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 21:37:16.372854 388 mini_oidc.cc:152] Starting JWKS server
I20260926 21:37:16.374778 388 webserver.cc:460] Webserver started at https://127.0.0.1:44491/ using document root <none> and password file <none>
I20260926 21:37:16.375047 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 21:37:16.375696 388 webserver.cc:460] Webserver started at http://127.0.0.1:37879/ using document root <none> and password file <none>
2026-09-26T21:37:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:16Z Disabled control of system clock
I20260926 21:37:16.401966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36323
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36323
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/testchainca.cert with env {}
W20260926 21:37:16.718830 5643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:16.719134 5643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:16.719224 5643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:16.726568 5643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:16.726697 5643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:16.726738 5643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:16.726796 5643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:16.726841 5643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 21:37:16.726897 5643 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 21:37:16.726929 5643 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44491/jwks.json
I20260926 21:37:16.737277 5643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:16.738624 5643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:16.739920 5643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:16.751969 5651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:16.752012 5649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:16.751994 5648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:16.752925 5643 server_base.cc:1034] running on GCE node
I20260926 21:37:16.754052 5643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:16.755558 5643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:16.756789 5643 hybrid_clock.cc:648] HybridClock initialized: now 1790458636756735 us; error 76 us; skew 500 ppm
I20260926 21:37:16.757148 5643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:37:16.766461 5643 webserver.cc:460] Webserver started at http://127.0.97.62:39179/ using document root <none> and password file <none>
I20260926 21:37:16.767237 5643 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:16.767377 5643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:16.767716 5643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:16.770720 5643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "c4cdaafd062a41feba49d31d5b3a94c8"
format_stamp: "Formatted at 2026-09-26 21:37:16 on dist-test-slave-lhhq"
server_key: "8439127c5da254bba2031d6a6767ba4c"
server_key_iv: "d3478d127f0f51a85e40f70c7bb5fde9"
server_key_version: "encryptionkey@0"
I20260926 21:37:16.771548 5643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "c4cdaafd062a41feba49d31d5b3a94c8"
format_stamp: "Formatted at 2026-09-26 21:37:16 on dist-test-slave-lhhq"
server_key: "8439127c5da254bba2031d6a6767ba4c"
server_key_iv: "d3478d127f0f51a85e40f70c7bb5fde9"
server_key_version: "encryptionkey@0"
I20260926 21:37:16.776968 5643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:37:16.780778 5657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:16.782348 5643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:16.782563 5643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "c4cdaafd062a41feba49d31d5b3a94c8"
format_stamp: "Formatted at 2026-09-26 21:37:16 on dist-test-slave-lhhq"
server_key: "8439127c5da254bba2031d6a6767ba4c"
server_key_iv: "d3478d127f0f51a85e40f70c7bb5fde9"
server_key_version: "encryptionkey@0"
I20260926 21:37:16.782819 5643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:16.811898 5643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:16.812804 5643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:16.813102 5643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:16.831200 5643 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36323
I20260926 21:37:16.831224 5708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36323 every 8 connection(s)
I20260926 21:37:16.832578 5643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:16.836329 5709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:16.840703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 5643
I20260926 21:37:16.840938 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:16.841605 388 external_mini_cluster.cc:1428] Setting key ae13385677887e91882937404d4d9066
I20260926 21:37:16.847102 5709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8: Bootstrap starting.
I20260926 21:37:16.850664 5709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:16.851835 5709 log.cc:824] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:16.855113 5709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8: No bootstrap required, opened a new log
I20260926 21:37:16.855581 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:16.843547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.0.97.62:36323)
0926 21:37:16.844560 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:16.844571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:16.844611 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:37:16.846311 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:16.846334 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:16.846782 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:16.847103 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:16.847675 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.848948 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.850825 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.851540 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.851683 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:16.853018 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:16.853034 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:16.853053 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:16.853706 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:16.853780 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:16.853801 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:16.853896 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:16.854169 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 21:37:16.854543 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":382,"threads_started":1}
I20260926 21:37:16.859839 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:37:16.864010 5709 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cdaafd062a41feba49d31d5b3a94c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36323 } }
I20260926 21:37:16.864392 5709 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:16.864521 5709 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cdaafd062a41feba49d31d5b3a94c8, State: Initialized, Role: FOLLOWER
I20260926 21:37:16.865146 5709 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cdaafd062a41feba49d31d5b3a94c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36323 } }
I20260926 21:37:16.865435 5709 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:16.865584 5709 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:16.865760 5709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:16.868050 5709 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cdaafd062a41feba49d31d5b3a94c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36323 } }
I20260926 21:37:16.868566 5709 leader_election.cc:302] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4cdaafd062a41feba49d31d5b3a94c8; no voters:
I20260926 21:37:16.869172 5709 leader_election.cc:288] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:16.869390 5714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:16.870146 5714 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [term 1 LEADER]: Becoming Leader. State: Replica: c4cdaafd062a41feba49d31d5b3a94c8, State: Running, Role: LEADER
I20260926 21:37:16.870769 5714 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cdaafd062a41feba49d31d5b3a94c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36323 } }
I20260926 21:37:16.871459 5709 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:16.873479 5716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4cdaafd062a41feba49d31d5b3a94c8. Latest consensus state: current_term: 1 leader_uuid: "c4cdaafd062a41feba49d31d5b3a94c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cdaafd062a41feba49d31d5b3a94c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36323 } } }
I20260926 21:37:16.873955 5716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:16.874847 5720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:16.874820 5715 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4cdaafd062a41feba49d31d5b3a94c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cdaafd062a41feba49d31d5b3a94c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36323 } } }
I20260926 21:37:16.875281 5715 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:16.884539 5720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:16.894063 5720 catalog_manager.cc:1347] Generated new cluster ID: 56b04aaa48bd4369a7588b0f2c33f03e
I20260926 21:37:16.894230 5720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:16.913612 5720 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:16.915079 5720 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:16.923147 5720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4cdaafd062a41feba49d31d5b3a94c8: Generated new TSK 0
I20260926 21:37:16.924036 5720 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 21:37:16.957862 5739 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 21:37:16.958002 5739 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 21:37:16.959314 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:16.938629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.0.97.62:36323)
0926 21:37:16.938829 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:16.938835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:16.939178 (+ 343us) server_negotiation.cc:407] Connection header received
0926 21:37:16.939403 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:16.939409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:16.939504 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:16.939686 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0926 21:37:16.940213 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.941389 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.942872 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.943584 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.943714 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:16.944280 (+ 566us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 21:37:16.958585 (+ 14305us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 21:37:16.959020 (+ 435us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 21:37:16.959468 5712 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 21:37:16.959460 5738 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:16.938460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36323 (local address 127.0.0.1:34668)
0926 21:37:16.939075 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:16.939090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:16.939209 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:16.939758 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:16.939762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:16.939773 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0926 21:37:16.940051 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:16.940059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.941575 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.941579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:16.942720 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:16.942732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:16.943750 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:16.943812 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:16.943948 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:16.943985 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 21:37:16.958888 (+ 14903us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 21:37:16.959121 (+ 233us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36323: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":335,"threads_started":1}
I20260926 21:37:16.960878 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 5643
I20260926 21:37:16.962124 5704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:17.054656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 5643
2026-09-26T21:37:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (735 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 21:37:17.108428 388 mini_oidc.cc:152] Starting JWKS server
I20260926 21:37:17.110431 388 webserver.cc:460] Webserver started at https://127.0.0.1:35505/ using document root <none> and password file <none>
I20260926 21:37:17.110678 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 21:37:17.111354 388 webserver.cc:460] Webserver started at http://127.0.0.1:33197/ using document root <none> and password file <none>
2026-09-26T21:37:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:17Z Disabled control of system clock
I20260926 21:37:17.138271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46109
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35505/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 21:37:17.461628 5746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:17.461923 5746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:17.462009 5746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:17.469174 5746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:17.469304 5746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:17.469341 5746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:17.469398 5746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:17.469439 5746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 21:37:17.469493 5746 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 21:37:17.469532 5746 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35505/jwks.json
I20260926 21:37:17.478595 5746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35505/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:17.479866 5746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:17.481568 5746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:17.494202 5751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:17.494231 5754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:17.494278 5752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:17.495157 5746 server_base.cc:1034] running on GCE node
I20260926 21:37:17.495887 5746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:17.497356 5746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:17.498579 5746 hybrid_clock.cc:648] HybridClock initialized: now 1790458637498545 us; error 44 us; skew 500 ppm
I20260926 21:37:17.498927 5746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:37:17.501072 5746 webserver.cc:460] Webserver started at http://127.0.97.62:41715/ using document root <none> and password file <none>
I20260926 21:37:17.501812 5746 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:17.501950 5746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:17.502286 5746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:17.505311 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7"
format_stamp: "Formatted at 2026-09-26 21:37:17 on dist-test-slave-lhhq"
server_key: "299a0723c03b9501815a57b38d7c4be4"
server_key_iv: "4a0f448f557a217cb4e67dc94283fda7"
server_key_version: "encryptionkey@0"
I20260926 21:37:17.506124 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7"
format_stamp: "Formatted at 2026-09-26 21:37:17 on dist-test-slave-lhhq"
server_key: "299a0723c03b9501815a57b38d7c4be4"
server_key_iv: "4a0f448f557a217cb4e67dc94283fda7"
server_key_version: "encryptionkey@0"
I20260926 21:37:17.511161 5746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:37:17.514662 5760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:17.516003 5746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:17.516184 5746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7"
format_stamp: "Formatted at 2026-09-26 21:37:17 on dist-test-slave-lhhq"
server_key: "299a0723c03b9501815a57b38d7c4be4"
server_key_iv: "4a0f448f557a217cb4e67dc94283fda7"
server_key_version: "encryptionkey@0"
I20260926 21:37:17.516399 5746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:17.546624 5746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:17.547536 5746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:17.547773 5746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:17.566102 5746 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46109
I20260926 21:37:17.566285 5811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46109 every 8 connection(s)
I20260926 21:37:17.567865 5746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:17.573668 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:17.578145 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 5746
I20260926 21:37:17.578353 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:17.579217 388 external_mini_cluster.cc:1428] Setting key 03b02d09ea11bf2bab707d99a75661ce
I20260926 21:37:17.589746 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7: Bootstrap starting.
I20260926 21:37:17.593801 5812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:17.595228 5812 log.cc:824] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:17.596084 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:17.581155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.0.97.62:46109)
0926 21:37:17.582510 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:17.582527 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:17.582575 (+ 48us) server_negotiation.cc:407] Connection header received
0926 21:37:17.584779 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:17.584808 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:17.585493 (+ 685us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:17.585973 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:17.586595 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:17.588523 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:17.590019 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:17.591005 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:17.591184 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:17.592941 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:17.592955 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:17.592973 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:17.593801 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:17.593926 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:17.593956 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:17.594072 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:17.594292 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 21:37:17.594636 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":551,"threads_started":1}
I20260926 21:37:17.598913 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7: No bootstrap required, opened a new log
I20260926 21:37:17.600968 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:37:17.609850 5812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 21:37:17.610349 5812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:17.610464 5812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a35f0e9128f44b7b48a3a34a6ec34c7, State: Initialized, Role: FOLLOWER
I20260926 21:37:17.611199 5812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 21:37:17.611455 5812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:17.611598 5812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:17.611793 5812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:17.614672 5812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 21:37:17.615207 5812 leader_election.cc:302] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a35f0e9128f44b7b48a3a34a6ec34c7; no voters:
I20260926 21:37:17.615798 5812 leader_election.cc:288] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:17.615942 5817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:17.617002 5817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [term 1 LEADER]: Becoming Leader. State: Replica: 3a35f0e9128f44b7b48a3a34a6ec34c7, State: Running, Role: LEADER
I20260926 21:37:17.617547 5817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 21:37:17.617856 5812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:17.620229 5819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a35f0e9128f44b7b48a3a34a6ec34c7. Latest consensus state: current_term: 1 leader_uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260926 21:37:17.620607 5819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:17.621160 5818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a35f0e9128f44b7b48a3a34a6ec34c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260926 21:37:17.621515 5818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:17.621817 5824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:17.628206 5824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:17.637148 5824 catalog_manager.cc:1347] Generated new cluster ID: 56676b992fa7420b94d9c70f37e151ad
I20260926 21:37:17.637310 5824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:17.689643 5824 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:17.691514 5824 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:17.710078 5824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a35f0e9128f44b7b48a3a34a6ec34c7: Generated new TSK 0
I20260926 21:37:17.711376 5824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:37:17.744515 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:17.735769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.0.97.62:46109)
0926 21:37:17.736044 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:17.736051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:17.736450 (+ 399us) server_negotiation.cc:407] Connection header received
0926 21:37:17.736688 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:17.736692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:17.736770 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:17.736916 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0926 21:37:17.737522 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:17.738269 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:17.740348 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:17.740785 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:17.740891 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:17.741519 (+ 628us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 21:37:17.743897 (+ 2378us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 21:37:17.744212 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":93}
W20260926 21:37:17.744638 5815 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 21:37:17.746759 5842 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:17.735618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.0.1:53738)
0926 21:37:17.736334 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:17.736355 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:17.736515 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:17.736957 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:17.736963 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:17.736977 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 21:37:17.737349 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:17.737363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:17.738413 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:17.738421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:17.740180 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:17.740197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:17.740908 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:17.740986 (+ 78us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:17.741181 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:17.741258 (+ 77us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 21:37:17.744116 (+ 2858us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 21:37:17.746173 (+ 2057us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:46109: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":1727,"thread_start_us":385,"threads_started":1}
I20260926 21:37:17.748948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 5746
I20260926 21:37:17.750108 5807 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 21:37:17.756604 5843 webserver.cc:533] Webserver: sslize failed:
E20260926 21:37:17.756700 5843 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 21:37:17.844065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 5746
2026-09-26T21:37:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (790 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T21:37:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:17Z Disabled control of system clock
I20260926 21:37:17.925796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45045
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45045
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/testchainca.cert with env {}
W20260926 21:37:18.245962 5848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:18.246270 5848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:18.246445 5848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:18.253721 5848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:18.253849 5848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:18.253916 5848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:18.254015 5848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:18.254096 5848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 21:37:18.254194 5848 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 21:37:18.254254 5848 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 21:37:18.264312 5848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:18.265697 5848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:18.266984 5848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:18.278741 5854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:18.278772 5856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:18.278750 5853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:18.279284 5848 server_base.cc:1034] running on GCE node
I20260926 21:37:18.280128 5848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:18.281672 5848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:18.282910 5848 hybrid_clock.cc:648] HybridClock initialized: now 1790458638282856 us; error 48 us; skew 500 ppm
I20260926 21:37:18.283280 5848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:37:18.285653 5848 webserver.cc:460] Webserver started at http://127.0.97.62:34859/ using document root <none> and password file <none>
I20260926 21:37:18.286345 5848 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:18.286484 5848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:18.286787 5848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:18.289784 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "309d1b9ca571450c94a3b048b113682f"
format_stamp: "Formatted at 2026-09-26 21:37:18 on dist-test-slave-lhhq"
server_key: "7d5706519434c9aaf298596c7aae9587"
server_key_iv: "b14c3ff3bbc98e7c66fdda5e86a3e860"
server_key_version: "encryptionkey@0"
I20260926 21:37:18.290571 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "309d1b9ca571450c94a3b048b113682f"
format_stamp: "Formatted at 2026-09-26 21:37:18 on dist-test-slave-lhhq"
server_key: "7d5706519434c9aaf298596c7aae9587"
server_key_iv: "b14c3ff3bbc98e7c66fdda5e86a3e860"
server_key_version: "encryptionkey@0"
I20260926 21:37:18.295751 5848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:37:18.299357 5862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:18.300694 5848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:37:18.300873 5848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "309d1b9ca571450c94a3b048b113682f"
format_stamp: "Formatted at 2026-09-26 21:37:18 on dist-test-slave-lhhq"
server_key: "7d5706519434c9aaf298596c7aae9587"
server_key_iv: "b14c3ff3bbc98e7c66fdda5e86a3e860"
server_key_version: "encryptionkey@0"
I20260926 21:37:18.301093 5848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:18.312991 5848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:18.313939 5848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:18.314214 5848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:18.332024 5848 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45045
I20260926 21:37:18.332046 5913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45045 every 8 connection(s)
I20260926 21:37:18.333407 5848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:18.335238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 5848
I20260926 21:37:18.335443 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:18.336093 388 external_mini_cluster.cc:1428] Setting key 577d2c7bbe1ee380d8b273465084bfad
I20260926 21:37:18.337628 5914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:18.351538 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:18.338064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.97.62:45045)
0926 21:37:18.339074 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:18.339089 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:18.339144 (+ 55us) server_negotiation.cc:407] Connection header received
0926 21:37:18.340965 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:18.340995 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:18.341678 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:18.342161 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:18.342720 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:18.344766 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:18.346170 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:18.347173 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:18.347356 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:18.348501 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:18.348516 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:18.348536 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:37:18.349348 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:18.349446 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:37:18.349472 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:18.349603 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:18.349805 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 21:37:18.350180 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":290,"threads_started":1}
I20260926 21:37:18.352592 5914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f: Bootstrap starting.
I20260926 21:37:18.355609 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:37:18.356137 5914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:18.357374 5914 log.cc:824] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:18.360844 5914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f: No bootstrap required, opened a new log
I20260926 21:37:18.368413 5914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309d1b9ca571450c94a3b048b113682f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } }
I20260926 21:37:18.368813 5914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:18.368932 5914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309d1b9ca571450c94a3b048b113682f, State: Initialized, Role: FOLLOWER
I20260926 21:37:18.369551 5914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309d1b9ca571450c94a3b048b113682f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } }
I20260926 21:37:18.369801 5914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:18.369939 5914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:18.370123 5914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:18.372520 5914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309d1b9ca571450c94a3b048b113682f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } }
I20260926 21:37:18.373239 5914 leader_election.cc:302] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 309d1b9ca571450c94a3b048b113682f; no voters:
I20260926 21:37:18.373907 5914 leader_election.cc:288] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:18.374082 5919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:18.374851 5919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [term 1 LEADER]: Becoming Leader. State: Replica: 309d1b9ca571450c94a3b048b113682f, State: Running, Role: LEADER
I20260926 21:37:18.375417 5919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309d1b9ca571450c94a3b048b113682f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } }
I20260926 21:37:18.376091 5914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:18.378530 5920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "309d1b9ca571450c94a3b048b113682f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309d1b9ca571450c94a3b048b113682f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } } }
I20260926 21:37:18.379151 5920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:18.378572 5921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 309d1b9ca571450c94a3b048b113682f. Latest consensus state: current_term: 1 leader_uuid: "309d1b9ca571450c94a3b048b113682f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309d1b9ca571450c94a3b048b113682f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } } }
I20260926 21:37:18.379765 5921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:18.381018 5928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:18.386724 5928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:18.397316 5928 catalog_manager.cc:1347] Generated new cluster ID: 15253fcc0e044b17aa01357cfd48cdb1
I20260926 21:37:18.397548 5928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:18.418985 5928 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:18.420912 5928 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:18.438150 5928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 309d1b9ca571450c94a3b048b113682f: Generated new TSK 0
I20260926 21:37:18.439291 5928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:37:18.575019 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:18.465416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.97.62:45045)
0926 21:37:18.465666 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:18.465673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:18.466038 (+ 365us) server_negotiation.cc:407] Connection header received
0926 21:37:18.466301 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:18.466305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:18.466373 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:18.466509 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0926 21:37:18.467046 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:18.467741 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:18.469921 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:18.470701 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:18.470819 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:18.471458 (+ 639us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 21:37:18.574282 (+102824us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 21:37:18.574709 (+ 427us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48268: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 21:37:18.575160 5917 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48268: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 21:37:18.575171 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:18.465287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45045 (local address 127.0.0.1:48268)
0926 21:37:18.465915 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:18.465936 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:18.466098 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:18.466555 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:18.466560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:18.466574 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 21:37:18.466898 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:18.466909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:18.467888 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:18.467894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:18.469693 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:18.469708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:18.470838 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:18.470898 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:18.471102 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:18.471153 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 21:37:18.574597 (+103444us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 21:37:18.574824 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45045: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":323,"threads_started":1}
I20260926 21:37:18.576613 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 5848
I20260926 21:37:18.577910 5909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:18.679296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 5848
2026-09-26T21:37:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (814 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (405 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (405 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:37:19 dist-test-slave-lhhq krb5kdc[5958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:37:19 dist-test-slave-lhhq krb5kdc[5958](info): set up 2 sockets
Sep 26 21:37:19 dist-test-slave-lhhq krb5kdc[5958](info): commencing operation
krb5kdc: starting...
W20260926 21:37:20.582268 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:37:20 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:37:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:37:20.789634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40311
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:21.107856 5971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:21.108146 5971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:21.108328 5971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:21.115638 5971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:21.115746 5971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:21.115864 5971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:21.115969 5971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:21.116053 5971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:21.125878 5971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:21.127218 5971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:21.128590 5971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:21.140143 5976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:21.140225 5977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:21.140177 5979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:21.141300 5971 server_base.cc:1034] running on GCE node
I20260926 21:37:21.142096 5971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:21.143558 5971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:21.144757 5971 hybrid_clock.cc:648] HybridClock initialized: now 1790458641144714 us; error 31 us; skew 500 ppm
I20260926 21:37:21.145097 5971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:21 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:21.148145 5971 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:37:21.149600 5971 webserver.cc:460] Webserver started at http://127.0.97.62:42665/ using document root <none> and password file <none>
I20260926 21:37:21.150240 5971 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:21.150347 5971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:21.150594 5971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:21.153553 5971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "abb0399011c146858aa40cee7fcdc99d"
format_stamp: "Formatted at 2026-09-26 21:37:21 on dist-test-slave-lhhq"
server_key: "788346522da7b3f504735110ff495b2f"
server_key_iv: "f7e8dbbde6b62193fa628ceb8cf11107"
server_key_version: "encryptionkey@0"
I20260926 21:37:21.154282 5971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "abb0399011c146858aa40cee7fcdc99d"
format_stamp: "Formatted at 2026-09-26 21:37:21 on dist-test-slave-lhhq"
server_key: "788346522da7b3f504735110ff495b2f"
server_key_iv: "f7e8dbbde6b62193fa628ceb8cf11107"
server_key_version: "encryptionkey@0"
I20260926 21:37:21.159399 5971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:37:21.162760 5986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:21.164041 5971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:37:21.164227 5971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "abb0399011c146858aa40cee7fcdc99d"
format_stamp: "Formatted at 2026-09-26 21:37:21 on dist-test-slave-lhhq"
server_key: "788346522da7b3f504735110ff495b2f"
server_key_iv: "f7e8dbbde6b62193fa628ceb8cf11107"
server_key_version: "encryptionkey@0"
I20260926 21:37:21.164429 5971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:21.187635 5971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:21.191515 5971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:21.191808 5971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:21.209628 5971 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40311
I20260926 21:37:21.209710 6037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40311 every 8 connection(s)
I20260926 21:37:21.211440 5971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:21.215904 6038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:21.219376 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 5971
I20260926 21:37:21.219610 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:21.220331 388 external_mini_cluster.cc:1428] Setting key 52a96c78078d99df2e597b3ad5637105
I20260926 21:37:21.229976 6038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d: Bootstrap starting.
I20260926 21:37:21.233446 6038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:21.234400 6038 log.cc:824] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:37:21 dist-test-slave-lhhq krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:21.237327 6038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d: No bootstrap required, opened a new log
I20260926 21:37:21.244854 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:21.222614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58562 (local address 127.0.97.62:40311)
0926 21:37:21.223469 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:21.223484 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:21.223530 (+ 46us) server_negotiation.cc:407] Connection header received
0926 21:37:21.225775 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:21.225805 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:21.226328 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:21.226793 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:21.228043 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:21.229999 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:21.231258 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:21.232316 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:21.232512 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:21.236361 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:21.236372 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:21.236389 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:21.236445 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:21.240448 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:21.241112 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:21.241120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:21.241129 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:21.241349 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:21.241743 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:21.241752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:21.241755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:21.242374 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:21.242671 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:21.243034 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 21:37:21.243484 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":416,"threads_started":1}
I20260926 21:37:21.247262 6038 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb0399011c146858aa40cee7fcdc99d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40311 } }
I20260926 21:37:21.247611 6038 raft_consensus.cc:374] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:21.247757 6038 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb0399011c146858aa40cee7fcdc99d, State: Initialized, Role: FOLLOWER
I20260926 21:37:21.248301 6038 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb0399011c146858aa40cee7fcdc99d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40311 } }
I20260926 21:37:21.248528 6038 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:21.248669 6038 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:21.248854 6038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:21.251032 6038 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb0399011c146858aa40cee7fcdc99d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40311 } }
I20260926 21:37:21.251498 6038 leader_election.cc:302] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb0399011c146858aa40cee7fcdc99d; no voters:
I20260926 21:37:21.252043 6038 leader_election.cc:288] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:21.252170 6043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:21.252825 6043 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [term 1 LEADER]: Becoming Leader. State: Replica: abb0399011c146858aa40cee7fcdc99d, State: Running, Role: LEADER
I20260926 21:37:21.253356 6043 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb0399011c146858aa40cee7fcdc99d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40311 } }
I20260926 21:37:21.254047 6038 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:21.256781 6045 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb0399011c146858aa40cee7fcdc99d. Latest consensus state: current_term: 1 leader_uuid: "abb0399011c146858aa40cee7fcdc99d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb0399011c146858aa40cee7fcdc99d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40311 } } }
I20260926 21:37:21.256935 6044 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abb0399011c146858aa40cee7fcdc99d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb0399011c146858aa40cee7fcdc99d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40311 } } }
I20260926 21:37:21.257174 6045 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:21.257236 6044 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:21.262365 6051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:21.269039 6051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:21.280129 6051 catalog_manager.cc:1347] Generated new cluster ID: 34676371eb3445da86b00a4aa62886f8
I20260926 21:37:21.280257 6051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:21.302028 6051 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:21.303321 6051 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:21.311717 6051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abb0399011c146858aa40cee7fcdc99d: Generated new TSK 0
I20260926 21:37:21.312769 6051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:37:21.366952 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40311
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:21.692402 6064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:21.692723 6064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:21.692909 6064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:21.700292 6064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:21.700436 6064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:21.700615 6064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:21.710160 6064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:21.711426 6064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:21.712700 6064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:21.727262 6069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:21.727703 6070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:21.727749 6072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:21.730207 6064 server_base.cc:1034] running on GCE node
I20260926 21:37:21.730973 6064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:21.731979 6064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:21.733271 6064 hybrid_clock.cc:648] HybridClock initialized: now 1790458641733141 us; error 66 us; skew 500 ppm
I20260926 21:37:21.733608 6064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:21 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:21.737067 6064 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:37:21.738572 6064 webserver.cc:460] Webserver started at http://127.0.97.1:45285/ using document root <none> and password file <none>
I20260926 21:37:21.739310 6064 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:21.739452 6064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:21.739764 6064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:21.742748 6064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "a173cbe6792f43f588bbc9f8cfe50627"
format_stamp: "Formatted at 2026-09-26 21:37:21 on dist-test-slave-lhhq"
server_key: "7bbcd7c7b86dc1f15e981e397314bb7a"
server_key_iv: "d0003bba1b9110dbaa3140951b5c1ce0"
server_key_version: "encryptionkey@0"
I20260926 21:37:21.743530 6064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a173cbe6792f43f588bbc9f8cfe50627"
format_stamp: "Formatted at 2026-09-26 21:37:21 on dist-test-slave-lhhq"
server_key: "7bbcd7c7b86dc1f15e981e397314bb7a"
server_key_iv: "d0003bba1b9110dbaa3140951b5c1ce0"
server_key_version: "encryptionkey@0"
I20260926 21:37:21.748617 6064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:37:21.752411 6079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:21.753808 6064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:21.753993 6064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "a173cbe6792f43f588bbc9f8cfe50627"
format_stamp: "Formatted at 2026-09-26 21:37:21 on dist-test-slave-lhhq"
server_key: "7bbcd7c7b86dc1f15e981e397314bb7a"
server_key_iv: "d0003bba1b9110dbaa3140951b5c1ce0"
server_key_version: "encryptionkey@0"
I20260926 21:37:21.754194 6064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:21.767326 6064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:21.776930 6064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:21.777302 6064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:21.778298 6064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:21.779664 6064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:21.779772 6064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:21.779922 6064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:21.779994 6064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:21.814745 6064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46563
I20260926 21:37:21.814852 6191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46563 every 8 connection(s)
I20260926 21:37:21.816036 6064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:21.817405 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 6064
I20260926 21:37:21.817619 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:37:21.818464 388 external_mini_cluster.cc:1428] Setting key 5196fded9247ebdb74b23413593e9150
Sep 26 21:37:21 dist-test-slave-lhhq krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:21.838948 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:21.819092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53699 (local address 127.0.97.62:40311)
0926 21:37:21.819852 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:21.819858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:21.820851 (+ 993us) server_negotiation.cc:407] Connection header received
0926 21:37:21.822738 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:21.822746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:21.822837 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:21.822960 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:21.824467 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:21.825689 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:21.827384 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:21.828165 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:21.828326 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:21.832019 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:21.832028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:21.832035 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:21.832083 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:21.835613 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:21.836354 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:21.836362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:21.836368 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:21.836521 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:21.836960 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:21.836965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:21.836968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:21.837391 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:21.837536 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:21.838256 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 21:37:21.838502 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":514,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:37:21.839696 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:21.819813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40311 (local address 127.0.97.1:53699)
0926 21:37:21.820677 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:21.820724 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:21.822459 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:21.823171 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:21.823180 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:21.823719 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:21.824291 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:21.824308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:21.825860 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:21.825868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:21.827240 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:21.827257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:21.828382 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:21.828387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:21.828599 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:21.829359 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:21.829382 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:21.831766 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:21.835763 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:21.835771 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:21.835785 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:21.836230 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:21.836669 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:21.836672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:21.836674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:21.836847 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:21.837521 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:21.837527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:21.837861 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 21:37:21.838101 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 21:37:21.838489 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":373,"threads_started":1}
I20260926 21:37:21.841645 6192 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:37:21.842007 6192 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:21.842792 6192 heartbeater.cc:511] Master 127.0.97.62:40311 requested a full tablet report, sending...
I20260926 21:37:21.845252 6003 ts_manager.cc:194] Registered new tserver with Master: a173cbe6792f43f588bbc9f8cfe50627 (127.0.97.1:46563)
I20260926 21:37:21.847380 6003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:37:21.868734 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40311
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:22.184751 6198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:22.185079 6198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:22.185272 6198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:22.192397 6198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:22.192551 6198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:22.192735 6198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:22.202559 6198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:22.203899 6198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:22.205216 6198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:22.219641 6206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:22.219564 6204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:22.219990 6203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:22.220211 6198 server_base.cc:1034] running on GCE node
I20260926 21:37:22.220912 6198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:22.222229 6198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:22.223551 6198 hybrid_clock.cc:648] HybridClock initialized: now 1790458642223459 us; error 84 us; skew 500 ppm
I20260926 21:37:22.223922 6198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:22.227626 6198 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:37:22.229164 6198 webserver.cc:460] Webserver started at http://127.0.97.2:44379/ using document root <none> and password file <none>
I20260926 21:37:22.229920 6198 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:22.230077 6198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:22.230381 6198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:22.233372 6198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "6da29147ffc04197aca7d49638d2d1c2"
format_stamp: "Formatted at 2026-09-26 21:37:22 on dist-test-slave-lhhq"
server_key: "b4b3f4892165ce88e9f2cc15a4f25219"
server_key_iv: "efc18941db3a940d7511e57ebb3d1c33"
server_key_version: "encryptionkey@0"
I20260926 21:37:22.234133 6198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6da29147ffc04197aca7d49638d2d1c2"
format_stamp: "Formatted at 2026-09-26 21:37:22 on dist-test-slave-lhhq"
server_key: "b4b3f4892165ce88e9f2cc15a4f25219"
server_key_iv: "efc18941db3a940d7511e57ebb3d1c33"
server_key_version: "encryptionkey@0"
I20260926 21:37:22.239355 6198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:37:22.242905 6213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:22.244163 6198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:37:22.244334 6198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "6da29147ffc04197aca7d49638d2d1c2"
format_stamp: "Formatted at 2026-09-26 21:37:22 on dist-test-slave-lhhq"
server_key: "b4b3f4892165ce88e9f2cc15a4f25219"
server_key_iv: "efc18941db3a940d7511e57ebb3d1c33"
server_key_version: "encryptionkey@0"
I20260926 21:37:22.244552 6198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:22.259640 6198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:22.263847 6198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:22.264230 6198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:22.265321 6198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:22.266685 6198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:22.266799 6198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:22.266927 6198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:22.267004 6198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:22.299963 6198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45069
I20260926 21:37:22.300002 6325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45069 every 8 connection(s)
I20260926 21:37:22.301347 6198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:22.308109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 6198
I20260926 21:37:22.308300 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:37:22.308830 388 external_mini_cluster.cc:1428] Setting key 9e99dea30b4fe4a2c3d8e63f8ed87833
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:22.322338 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:22.303819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39749 (local address 127.0.97.62:40311)
0926 21:37:22.304032 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:22.304039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:22.305226 (+ 1187us) server_negotiation.cc:407] Connection header received
0926 21:37:22.306939 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:22.306943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:22.307027 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:22.307119 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:22.308687 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.309700 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.311612 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.312425 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.312572 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:22.316020 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:22.316030 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:22.316037 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:22.316082 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:22.318988 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.319976 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.319980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.319982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.320103 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.320581 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.320585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.320588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.320905 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:22.321025 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:22.321931 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 21:37:22.322106 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:37:22.323939 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:22.304332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40311 (local address 127.0.97.2:39749)
0926 21:37:22.305050 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:22.305088 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:22.306686 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:22.307318 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:22.307326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:22.307850 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:22.308524 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:22.308543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.309861 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.309866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:22.311469 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:22.311481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.312625 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.312630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:22.312820 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:22.313503 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:22.313524 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:22.315794 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:22.319158 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:22.319171 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:22.319191 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:22.319853 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:22.320209 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:22.320216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:22.320222 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:22.320467 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:22.321044 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:22.321055 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:22.321564 (+ 509us) client_negotiation.cc:769] Sending connection context
0926 21:37:22.321870 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 21:37:22.322433 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":285,"threads_started":1}
I20260926 21:37:22.325464 6326 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40311
I20260926 21:37:22.325769 6326 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:22.326507 6326 heartbeater.cc:511] Master 127.0.97.62:40311 requested a full tablet report, sending...
I20260926 21:37:22.327934 6003 ts_manager.cc:194] Registered new tserver with Master: 6da29147ffc04197aca7d49638d2d1c2 (127.0.97.2:45069)
I20260926 21:37:22.328704 6003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39749
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:37:22.356796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40311
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:22.686525 6331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:22.686813 6331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:22.687019 6331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:22.694514 6331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:22.694625 6331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:22.694775 6331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:22.705579 6331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:22.706871 6331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:22.708173 6331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:22.721896 6336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:22.721895 6337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:22.722404 6339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:22.722754 6331 server_base.cc:1034] running on GCE node
I20260926 21:37:22.723433 6331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:22.724429 6331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:22.725682 6331 hybrid_clock.cc:648] HybridClock initialized: now 1790458642725611 us; error 64 us; skew 500 ppm
I20260926 21:37:22.726019 6331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:22.729102 6331 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:37:22.730840 6331 webserver.cc:460] Webserver started at http://127.0.97.3:32881/ using document root <none> and password file <none>
I20260926 21:37:22.731515 6331 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:22.731616 6331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:22.731886 6331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:22.735023 6331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "4043e4dff2104a75a0207a6c961d3403"
format_stamp: "Formatted at 2026-09-26 21:37:22 on dist-test-slave-lhhq"
server_key: "1f62eb28363fe711771fdeaff1fa48ea"
server_key_iv: "0eda11d22c2cf1cc76a960d8d01fa522"
server_key_version: "encryptionkey@0"
I20260926 21:37:22.735785 6331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4043e4dff2104a75a0207a6c961d3403"
format_stamp: "Formatted at 2026-09-26 21:37:22 on dist-test-slave-lhhq"
server_key: "1f62eb28363fe711771fdeaff1fa48ea"
server_key_iv: "0eda11d22c2cf1cc76a960d8d01fa522"
server_key_version: "encryptionkey@0"
I20260926 21:37:22.741330 6331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 21:37:22.744748 6346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:22.746130 6331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:22.746306 6331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "4043e4dff2104a75a0207a6c961d3403"
format_stamp: "Formatted at 2026-09-26 21:37:22 on dist-test-slave-lhhq"
server_key: "1f62eb28363fe711771fdeaff1fa48ea"
server_key_iv: "0eda11d22c2cf1cc76a960d8d01fa522"
server_key_version: "encryptionkey@0"
I20260926 21:37:22.746484 6331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:22.764696 6331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:22.769044 6331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:22.769366 6331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:22.770375 6331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:22.771780 6331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:22.771855 6331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:22.772043 6331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:22.772096 6331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:22.805096 6331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36473
I20260926 21:37:22.805145 6458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36473 every 8 connection(s)
I20260926 21:37:22.806443 6331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:22.807358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 6331
I20260926 21:37:22.807569 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:37:22.808102 388 external_mini_cluster.cc:1428] Setting key 3548c1021c15cd3b5d35f485dbd062c0
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:22.828312 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:22.809496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49243 (local address 127.0.97.62:40311)
0926 21:37:22.809695 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:22.809701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:22.811303 (+ 1602us) server_negotiation.cc:407] Connection header received
0926 21:37:22.813092 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:22.813096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:22.813160 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:22.813317 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:22.815387 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.816149 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.818546 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.819019 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.819127 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:22.822769 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:22.822774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:22.822777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:22.822811 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:22.825590 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.826291 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.826295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.826297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.826435 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.826821 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.826824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.826827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.827105 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:22.827220 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:22.827876 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 21:37:22.828050 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:37:22.829319 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:22.810241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40311 (local address 127.0.97.3:49243)
0926 21:37:22.811129 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:22.811184 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:22.812810 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:22.813530 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:22.813538 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:22.814333 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:22.815175 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:22.815204 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.816320 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.816325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:22.818359 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:22.818374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.819264 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.819271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:22.819572 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:22.820281 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:22.820309 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:22.822573 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:22.825716 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:22.825723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:22.825737 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:22.826191 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:22.826529 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:22.826532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:22.826534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:22.826701 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:22.827212 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:22.827218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:22.827540 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 21:37:22.827768 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 21:37:22.828149 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":380,"threads_started":1}
I20260926 21:37:22.830797 6459 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40311
I20260926 21:37:22.831113 6459 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:22.831840 6459 heartbeater.cc:511] Master 127.0.97.62:40311 requested a full tablet report, sending...
I20260926 21:37:22.833297 6003 ts_manager.cc:194] Registered new tserver with Master: 4043e4dff2104a75a0207a6c961d3403 (127.0.97.3:36473)
I20260926 21:37:22.834169 6003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49243
I20260926 21:37:22.836359 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:37:22.850687 6192 heartbeater.cc:503] Master 127.0.97.62:40311 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/test-admin.keytab.
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:22.862694 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 21:37:22.868040 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:22.865416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.0.97.1:46563)
0926 21:37:22.866192 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:22.866199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:22.866223 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:37:22.866282 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:22.866288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:22.866497 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:22.866666 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:22.867558 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52286: BlockingRecv error: recv got EOF from 127.0.0.1:52286 (error 108)
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":367,"threads_started":1}
W20260926 21:37:22.868086 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:22.869782 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:22.871214 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:37:22.886530 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:22.872845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.97.1:46563)
0926 21:37:22.873045 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:22.873049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:22.873153 (+ 104us) server_negotiation.cc:407] Connection header received
0926 21:37:22.873353 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:22.873357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:22.873416 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:22.873501 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:22.874516 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.875592 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.876775 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.877487 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.877592 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:22.881093 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:22.881104 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:22.881119 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:22.881157 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:22.883770 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.884427 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.884435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.884442 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.884568 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.884948 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.884952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.884954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.885406 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:22.885648 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:22.885962 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 21:37:22.886270 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 21:37:22.892486 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:22.890668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.0.97.1:46563)
0926 21:37:22.890883 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:22.890889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:22.891092 (+ 203us) server_negotiation.cc:407] Connection header received
0926 21:37:22.891287 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:22.891292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:22.891368 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:22.891489 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:22.892207 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52302: BlockingRecv error: recv got EOF from 127.0.0.1:52302 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 21:37:22.892573 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:22.894162 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:22.895365 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:37:22.909085 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:22.896544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.0.97.1:46563)
0926 21:37:22.896738 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:22.896744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:22.897025 (+ 281us) server_negotiation.cc:407] Connection header received
0926 21:37:22.897220 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:22.897229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:22.897284 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:22.897407 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:22.898284 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.899149 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.900319 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.900730 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.900875 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:22.904246 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:22.904251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:22.904254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:22.904293 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:22.906663 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.907307 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.907314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.907317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.907448 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.907809 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.907814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.907817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.908074 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:22.908339 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:22.908605 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 21:37:22.908804 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 21:37:22.913367 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:22.911656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.0.97.1:46563)
0926 21:37:22.911881 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:22.911887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:22.912062 (+ 175us) server_negotiation.cc:407] Connection header received
0926 21:37:22.912189 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:22.912192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:22.912264 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:22.912341 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:22.913091 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52330: BlockingRecv error: recv got EOF from 127.0.0.1:52330 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 21:37:22.913738 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790458593267894-388-0/krb5kdc/krb5cc)
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:22.915215 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:22.916479 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 21:37:22 dist-test-slave-lhhq krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:37:22.930794 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:22.917751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.0.97.1:46563)
0926 21:37:22.917991 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:22.917997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:22.918215 (+ 218us) server_negotiation.cc:407] Connection header received
0926 21:37:22.918392 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:22.918399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:22.918469 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:22.918569 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:22.919526 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.920252 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.921530 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.921948 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.922078 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:22.925640 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:22.925645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:22.925648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:22.925680 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:22.928135 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.928812 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.928824 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.928828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.928981 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.929404 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.929410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.929414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.929691 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:22.930014 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:22.930259 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 21:37:22.930481 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:37:22.932351 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 6064
I20260926 21:37:22.944142 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:22.933026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.0.97.1:46563)
0926 21:37:22.933269 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:22.933274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:22.933518 (+ 244us) server_negotiation.cc:407] Connection header received
0926 21:37:22.933635 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:22.933639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:22.933692 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:22.933766 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:22.934617 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.935270 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.936402 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:22.936802 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:22.936895 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:22.939313 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:22.939318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:22.939321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:22.939349 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:22.941838 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.942448 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.942452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.942454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.942577 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:22.942882 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:22.942885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:22.942887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:22.943131 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:22.943315 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:22.943607 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 21:37:22.943788 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:37:22.944700 6187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:23.028818 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 6064
I20260926 21:37:23.044864 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 6198
Sep 26 21:37:23 dist-test-slave-lhhq krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:37:23.061033 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:23.045813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.0.97.2:45069)
0926 21:37:23.046390 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:23.046397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:23.046423 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:37:23.046492 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:23.046498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:23.046712 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:23.046839 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:23.047760 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:23.048907 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:23.050436 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:23.051060 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:23.051160 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:23.054990 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:23.055001 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:23.055015 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:23.055047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:23.057968 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:23.058744 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:23.058753 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:23.058756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:23.058898 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:23.059329 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:23.059341 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:23.059344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:23.059704 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:23.060033 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:23.060400 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 21:37:23.060696 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":308,"threads_started":1}
I20260926 21:37:23.062397 6321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:23.144398 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 6198
I20260926 21:37:23.160318 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 6331
Sep 26 21:37:23 dist-test-slave-lhhq krb5kdc[5958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:37:23.176450 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:23.161393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.0.97.3:36473)
0926 21:37:23.162074 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:23.162081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:23.162109 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:37:23.162171 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:23.162177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:23.162383 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:23.162517 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:23.163526 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:23.164653 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:23.166232 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:23.166784 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:23.166890 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:23.171088 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:23.171101 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:23.171117 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:23.171156 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:23.174000 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:23.174670 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:23.174677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:23.174679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:23.174795 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:23.175109 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:23.175112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:23.175114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:23.175431 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:23.175657 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:23.175887 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 21:37:23.176143 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":370,"threads_started":1}
I20260926 21:37:23.177827 6454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:23.268599 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 6331
I20260926 21:37:23.284543 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 5971
I20260926 21:37:23.285769 6033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:23.387727 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 5971
2026-09-26T21:37:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3904 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:37:23 dist-test-slave-lhhq krb5kdc[6486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:37:23 dist-test-slave-lhhq krb5kdc[6486](info): set up 2 sockets
Sep 26 21:37:23 dist-test-slave-lhhq krb5kdc[6486](info): commencing operation
krb5kdc: starting...
W20260926 21:37:24.493997 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:37:24 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:37:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 21:37:24.701253 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34137
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:25.027977 6499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:25.028334 6499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:25.028508 6499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:25.035702 6499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:25.035851 6499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:25.035969 6499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:25.036069 6499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:25.036155 6499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:25.045919 6499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:25.047245 6499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:25.048535 6499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:25.060248 6504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:25.060261 6505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:25.060220 6507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:25.061496 6499 server_base.cc:1034] running on GCE node
I20260926 21:37:25.062356 6499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:25.063867 6499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:25.065106 6499 hybrid_clock.cc:648] HybridClock initialized: now 1790458645065060 us; error 46 us; skew 500 ppm
I20260926 21:37:25.065481 6499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:25 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458645, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:25.068560 6499 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 21:37:25.070170 6499 webserver.cc:460] Webserver started at http://127.0.97.62:36139/ using document root <none> and password file <none>
I20260926 21:37:25.070865 6499 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:25.071017 6499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:25.071321 6499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:25.074318 6499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "fcff42acfbd941d28b12e39fd93cd9ce"
format_stamp: "Formatted at 2026-09-26 21:37:25 on dist-test-slave-lhhq"
server_key: "80d45c55b93539798ccc42121878174e"
server_key_iv: "732f6199ffbb8e0cfc18ed5fa587da2c"
server_key_version: "encryptionkey@0"
I20260926 21:37:25.075088 6499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "fcff42acfbd941d28b12e39fd93cd9ce"
format_stamp: "Formatted at 2026-09-26 21:37:25 on dist-test-slave-lhhq"
server_key: "80d45c55b93539798ccc42121878174e"
server_key_iv: "732f6199ffbb8e0cfc18ed5fa587da2c"
server_key_version: "encryptionkey@0"
I20260926 21:37:25.080304 6499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:37:25.083791 6514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:25.085278 6499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:37:25.085460 6499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "fcff42acfbd941d28b12e39fd93cd9ce"
format_stamp: "Formatted at 2026-09-26 21:37:25 on dist-test-slave-lhhq"
server_key: "80d45c55b93539798ccc42121878174e"
server_key_iv: "732f6199ffbb8e0cfc18ed5fa587da2c"
server_key_version: "encryptionkey@0"
I20260926 21:37:25.085669 6499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:25.107671 6499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:25.112035 6499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:25.112327 6499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:25.131270 6499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34137
I20260926 21:37:25.131278 6565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34137 every 8 connection(s)
I20260926 21:37:25.132651 6499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:25.136400 6566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:25.142369 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 6499
I20260926 21:37:25.142625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:25.143268 388 external_mini_cluster.cc:1428] Setting key aafe767f931f1353a6e6683832523d64
I20260926 21:37:25.148829 6566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: Bootstrap starting.
I20260926 21:37:25.152720 6566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:25.153921 6566 log.cc:824] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:25.157065 6566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: No bootstrap required, opened a new log
Sep 26 21:37:25 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:25.167454 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:25.145524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.0.97.62:34137)
0926 21:37:25.146357 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:25.146374 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:25.146430 (+ 56us) server_negotiation.cc:407] Connection header received
0926 21:37:25.148877 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:25.148906 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:25.149404 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:25.149858 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:25.151111 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:25.153233 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:25.154572 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:25.155419 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:25.155603 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:25.159903 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:25.159914 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:25.159928 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:25.159963 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:25.163156 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:25.164043 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:25.164049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:25.164059 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:25.164211 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:25.164643 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:25.164646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:25.164648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:25.165283 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:25.165492 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:25.166023 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 21:37:25.166408 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":348,"threads_started":1}
I20260926 21:37:25.167326 6566 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } }
I20260926 21:37:25.167783 6566 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:25.167923 6566 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcff42acfbd941d28b12e39fd93cd9ce, State: Initialized, Role: FOLLOWER
I20260926 21:37:25.168551 6566 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } }
I20260926 21:37:25.168797 6566 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:25.168952 6566 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:25.169178 6566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:25.171675 6566 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } }
I20260926 21:37:25.172219 6566 leader_election.cc:302] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcff42acfbd941d28b12e39fd93cd9ce; no voters:
I20260926 21:37:25.172865 6566 leader_election.cc:288] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:25.173015 6571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:25.174602 6571 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 1 LEADER]: Becoming Leader. State: Replica: fcff42acfbd941d28b12e39fd93cd9ce, State: Running, Role: LEADER
I20260926 21:37:25.175190 6566 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:25.175127 6571 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } }
I20260926 21:37:25.178244 6573 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcff42acfbd941d28b12e39fd93cd9ce. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } }
I20260926 21:37:25.178607 6573 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:25.178336 6572 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } }
I20260926 21:37:25.178807 6572 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:25.181919 6579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:25.188398 6579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:25.198364 6579 catalog_manager.cc:1347] Generated new cluster ID: fc41cec6fcd54f518d5ad96e19200613
I20260926 21:37:25.198490 6579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:25.206599 6579 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:25.208590 6579 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:25.226893 6579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: Generated new TSK 0
I20260926 21:37:25.228060 6579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 21:37:25.290820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34137
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:25.619361 6592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:25.619711 6592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:25.619884 6592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:25.627329 6592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:25.627497 6592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:25.627673 6592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:25.637490 6592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:25.638803 6592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:25.640163 6592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:25.653560 6597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:25.653589 6598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:25.653517 6600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:25.654413 6592 server_base.cc:1034] running on GCE node
I20260926 21:37:25.655061 6592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:25.656002 6592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:25.657279 6592 hybrid_clock.cc:648] HybridClock initialized: now 1790458645657228 us; error 48 us; skew 500 ppm
I20260926 21:37:25.657593 6592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:25 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458645, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:25.660795 6592 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 21:37:25.662245 6592 webserver.cc:460] Webserver started at http://127.0.97.1:43539/ using document root <none> and password file <none>
I20260926 21:37:25.662895 6592 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:25.662993 6592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:25.663235 6592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:25.666230 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "9cab11a3ee124d9093478181be796360"
format_stamp: "Formatted at 2026-09-26 21:37:25 on dist-test-slave-lhhq"
server_key: "adaf504c7f2d1db9c989fc317c6dd8c8"
server_key_iv: "7ee89fae0940e413416ef816294624c1"
server_key_version: "encryptionkey@0"
I20260926 21:37:25.667176 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9cab11a3ee124d9093478181be796360"
format_stamp: "Formatted at 2026-09-26 21:37:25 on dist-test-slave-lhhq"
server_key: "adaf504c7f2d1db9c989fc317c6dd8c8"
server_key_iv: "7ee89fae0940e413416ef816294624c1"
server_key_version: "encryptionkey@0"
I20260926 21:37:25.672237 6592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:37:25.675910 6607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:25.677495 6592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:37:25.677645 6592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "9cab11a3ee124d9093478181be796360"
format_stamp: "Formatted at 2026-09-26 21:37:25 on dist-test-slave-lhhq"
server_key: "adaf504c7f2d1db9c989fc317c6dd8c8"
server_key_iv: "7ee89fae0940e413416ef816294624c1"
server_key_version: "encryptionkey@0"
I20260926 21:37:25.677832 6592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:25.692436 6592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:25.696552 6592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:25.696810 6592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:25.698083 6592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:25.699570 6592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:25.699645 6592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:25.699791 6592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:25.699858 6592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:25.738968 6592 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46237
I20260926 21:37:25.739081 6719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46237 every 8 connection(s)
I20260926 21:37:25.740306 6592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:25.743196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 6592
I20260926 21:37:25.743449 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:37:25.744127 388 external_mini_cluster.cc:1428] Setting key 87857a6655073793e3a3d61b5647f2e2
Sep 26 21:37:25 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458645, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:25.763156 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:25.743110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45581 (local address 127.0.97.62:34137)
0926 21:37:25.743580 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:25.743586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:25.744758 (+ 1172us) server_negotiation.cc:407] Connection header received
0926 21:37:25.747093 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:25.747098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:25.747167 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:25.747260 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:25.749471 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:25.750210 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:25.752815 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:25.753347 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:25.753490 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:25.757376 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:25.757381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:25.757385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:25.757422 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:25.759922 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:25.760723 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:25.760728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:25.760730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:25.760846 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:25.761353 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:25.761357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:25.761362 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:25.761637 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:25.761790 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:25.762551 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 21:37:25.762787 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260926 21:37:25.764243 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:25.743717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.1:45581)
0926 21:37:25.744563 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:25.744612 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:25.746766 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:25.747562 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:25.747577 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:25.748414 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:25.749278 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:25.749306 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:25.750383 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:25.750392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:25.752627 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:25.752645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:25.753607 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:25.753615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:25.753858 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:25.754604 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:25.754627 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:25.757074 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:25.760112 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:25.760122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:25.760145 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:25.760578 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:25.760958 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:25.760964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:25.760967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:25.761178 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:25.761770 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:25.761778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:25.762158 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 21:37:25.762433 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 21:37:25.763010 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:37:25.766024 6720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34137
I20260926 21:37:25.766387 6720 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:25.767150 6720 heartbeater.cc:511] Master 127.0.97.62:34137 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260926 21:37:25.769776 6531 ts_manager.cc:194] Registered new tserver with Master: 9cab11a3ee124d9093478181be796360 (127.0.97.1:46237)
I20260926 21:37:25.771955 6531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 21:37:25.797255 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34137
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:26.119436 6726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:26.119757 6726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:26.119933 6726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:26.127288 6726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:26.127411 6726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:26.127615 6726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:26.137568 6726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:26.138895 6726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:26.140209 6726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:26.153764 6731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:26.153757 6732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:26.154068 6734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:26.154625 6726 server_base.cc:1034] running on GCE node
I20260926 21:37:26.155249 6726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:26.156293 6726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:26.157513 6726 hybrid_clock.cc:648] HybridClock initialized: now 1790458646157457 us; error 50 us; skew 500 ppm
I20260926 21:37:26.157794 6726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:26 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:26.160955 6726 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 21:37:26.162459 6726 webserver.cc:460] Webserver started at http://127.0.97.2:44869/ using document root <none> and password file <none>
I20260926 21:37:26.163128 6726 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:26.163230 6726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:26.163501 6726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:26.166427 6726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "df507c56b90747d89c0d1dd4c88b99cb"
format_stamp: "Formatted at 2026-09-26 21:37:26 on dist-test-slave-lhhq"
server_key: "0352b2a41d4d9fd0d8798836c8cd5221"
server_key_iv: "3daba643fcd5b12e43ef1d0087cc1b01"
server_key_version: "encryptionkey@0"
I20260926 21:37:26.167150 6726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "df507c56b90747d89c0d1dd4c88b99cb"
format_stamp: "Formatted at 2026-09-26 21:37:26 on dist-test-slave-lhhq"
server_key: "0352b2a41d4d9fd0d8798836c8cd5221"
server_key_iv: "3daba643fcd5b12e43ef1d0087cc1b01"
server_key_version: "encryptionkey@0"
I20260926 21:37:26.172338 6726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:37:26.175978 6741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:26.177538 6726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:37:26.177726 6726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "df507c56b90747d89c0d1dd4c88b99cb"
format_stamp: "Formatted at 2026-09-26 21:37:26 on dist-test-slave-lhhq"
server_key: "0352b2a41d4d9fd0d8798836c8cd5221"
server_key_iv: "3daba643fcd5b12e43ef1d0087cc1b01"
server_key_version: "encryptionkey@0"
I20260926 21:37:26.177927 6726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:26.196237 6726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:26.200380 6726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:26.200676 6726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:26.201683 6726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:26.203015 6726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:26.203088 6726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:26.203248 6726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:26.203310 6726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:26.236593 6726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36955
I20260926 21:37:26.236689 6853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36955 every 8 connection(s)
I20260926 21:37:26.238178 6726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:26.247790 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 6726
I20260926 21:37:26.248001 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:37:26.248451 388 external_mini_cluster.cc:1428] Setting key 2978988e3767b5faf253a21ce2e7780b
Sep 26 21:37:26 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:26.260445 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:26.241317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59101 (local address 127.0.97.62:34137)
0926 21:37:26.241511 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:26.241518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:26.242796 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 21:37:26.244498 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:26.244502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:26.244570 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:26.244659 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:26.246300 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:26.246961 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:26.248766 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:26.249524 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:26.249694 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:26.253134 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:26.253142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:26.253150 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:26.253222 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:26.256910 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:26.258003 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:26.258007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:26.258009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:26.258132 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:26.258663 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:26.258666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:26.258668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:26.258923 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:26.259024 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:26.260010 (+ 986us) server_negotiation.cc:299] Negotiation successful
0926 21:37:26.260190 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:37:26.262065 6856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:26.241838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.2:59101)
0926 21:37:26.242636 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:26.242678 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:26.244258 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:26.244932 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:26.244941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:26.245550 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:26.246137 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:26.246155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:26.247107 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:26.247111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:26.248603 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:26.248619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:26.249767 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:26.249772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:26.249990 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:26.250690 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:26.250731 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:26.252894 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:26.257107 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:26.257120 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:26.257142 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:26.257879 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:26.258245 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:26.258251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:26.258261 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:26.258546 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:26.259048 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:26.259060 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:26.259577 (+ 517us) client_negotiation.cc:769] Sending connection context
0926 21:37:26.259927 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 21:37:26.260516 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":303,"threads_started":1}
I20260926 21:37:26.263631 6854 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34137
I20260926 21:37:26.263993 6854 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:26.264885 6854 heartbeater.cc:511] Master 127.0.97.62:34137 requested a full tablet report, sending...
I20260926 21:37:26.266456 6531 ts_manager.cc:194] Registered new tserver with Master: df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955)
I20260926 21:37:26.267373 6531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59101
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 21:37:26.298054 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34137
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:26.624348 6859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:26.624698 6859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:26.624881 6859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:26.632182 6859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:26.632355 6859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:26.632548 6859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:26.642340 6859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:26.643704 6859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:26.644979 6859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:26.658090 6865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:26.658350 6867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:26.658694 6864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:26.659135 6859 server_base.cc:1034] running on GCE node
I20260926 21:37:26.659823 6859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:26.660773 6859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:26.662010 6859 hybrid_clock.cc:648] HybridClock initialized: now 1790458646661957 us; error 45 us; skew 500 ppm
I20260926 21:37:26.662335 6859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:26 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:26.665627 6859 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 21:37:26.667060 6859 webserver.cc:460] Webserver started at http://127.0.97.3:42521/ using document root <none> and password file <none>
I20260926 21:37:26.667829 6859 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:26.668000 6859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:26.668308 6859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:26.671271 6859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "a2139015a1b045adabf85d205381c5af"
format_stamp: "Formatted at 2026-09-26 21:37:26 on dist-test-slave-lhhq"
server_key: "ddc9b07b7c04497d37152a176bf97396"
server_key_iv: "83dd8ac481fe637dbb0b9503f0edc361"
server_key_version: "encryptionkey@0"
I20260926 21:37:26.672046 6859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a2139015a1b045adabf85d205381c5af"
format_stamp: "Formatted at 2026-09-26 21:37:26 on dist-test-slave-lhhq"
server_key: "ddc9b07b7c04497d37152a176bf97396"
server_key_iv: "83dd8ac481fe637dbb0b9503f0edc361"
server_key_version: "encryptionkey@0"
I20260926 21:37:26.677467 6859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 21:37:26.680734 6874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:26.682207 6859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:26.682382 6859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "a2139015a1b045adabf85d205381c5af"
format_stamp: "Formatted at 2026-09-26 21:37:26 on dist-test-slave-lhhq"
server_key: "ddc9b07b7c04497d37152a176bf97396"
server_key_iv: "83dd8ac481fe637dbb0b9503f0edc361"
server_key_version: "encryptionkey@0"
I20260926 21:37:26.682622 6859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:26.709372 6859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:26.713562 6859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:26.713858 6859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:26.714807 6859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:26.716150 6859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:26.716252 6859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:26.716403 6859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:26.716475 6859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:26.753014 6859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35619
I20260926 21:37:26.753070 6986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35619 every 8 connection(s)
I20260926 21:37:26.754359 6859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:26.758877 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 6859
I20260926 21:37:26.759073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:37:26.759500 388 external_mini_cluster.cc:1428] Setting key f7e39a51562e63571d3f003d41d359bc
Sep 26 21:37:26 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458646, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:26.774844 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:26.756976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52093 (local address 127.0.97.62:34137)
0926 21:37:26.757153 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:26.757157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:26.758311 (+ 1154us) server_negotiation.cc:407] Connection header received
0926 21:37:26.760403 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:26.760409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:26.760501 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:26.760623 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:26.762438 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:26.763392 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:26.764995 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:26.765490 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:26.765623 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:26.769103 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:26.769108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:26.769111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:26.769145 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:26.771641 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:26.772435 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:26.772448 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:26.772451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:26.772578 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:26.773026 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:26.773041 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:26.773043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:26.773374 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:26.773554 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:26.774266 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 21:37:26.774483 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:37:26.775823 6720 heartbeater.cc:503] Master 127.0.97.62:34137 was elected leader, sending a full tablet report...
I20260926 21:37:26.776221 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:26.757451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.3:52093)
0926 21:37:26.758142 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:26.758184 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:26.760090 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:26.760869 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:26.760882 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:26.761559 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:26.762137 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:26.762156 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:26.763555 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:26.763560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:26.764832 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:26.764845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:26.765722 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:26.765729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:26.765981 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:26.766659 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:26.766682 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:26.768851 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:26.771820 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:26.771830 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:26.771852 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:26.772298 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:26.772697 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:26.772702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:26.772706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:26.772897 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:26.773509 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:26.773517 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:26.773890 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 21:37:26.774163 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 21:37:26.774725 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":264,"threads_started":1}
I20260926 21:37:26.777943 6987 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34137
I20260926 21:37:26.778384 6987 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:26.779235 6987 heartbeater.cc:511] Master 127.0.97.62:34137 requested a full tablet report, sending...
I20260926 21:37:26.780881 6531 ts_manager.cc:194] Registered new tserver with Master: a2139015a1b045adabf85d205381c5af (127.0.97.3:35619)
I20260926 21:37:26.782014 6531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52093
I20260926 21:37:26.788143 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260926 21:37:26.836585 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42509
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34137,127.0.97.61:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:27.169457 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:27.169760 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:27.169854 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:27.177448 6992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:27.177559 6992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:27.177603 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:27.177659 6992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:27.177695 6992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:27.188259 6992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34137,127.0.97.61:42509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
W20260926 21:37:27.188577 6992 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:34137,127.0.97.61:42509'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 21:37:27.189685 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:27.191008 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:27.202921 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:27.203013 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:27.202958 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:27.204221 6992 server_base.cc:1034] running on GCE node
I20260926 21:37:27.205226 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:27.206851 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:27.208098 6992 hybrid_clock.cc:648] HybridClock initialized: now 1790458647208035 us; error 41 us; skew 500 ppm
I20260926 21:37:27.208472 6992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:27 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:27.219097 6992 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 21:37:27.220700 6992 webserver.cc:460] Webserver started at http://127.0.97.61:40317/ using document root <none> and password file <none>
I20260926 21:37:27.221520 6992 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:27.221652 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:27.221966 6992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:27.224869 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data/instance:
uuid: "dfe9f0e1bfd544e297a4fe182b445c02"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "ed368d98a9664865739168af95afb16d"
server_key_iv: "80d481396a716ff6d6c9b3a0f2fa1253"
server_key_version: "encryptionkey@0"
I20260926 21:37:27.225718 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal/instance:
uuid: "dfe9f0e1bfd544e297a4fe182b445c02"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "ed368d98a9664865739168af95afb16d"
server_key_iv: "80d481396a716ff6d6c9b3a0f2fa1253"
server_key_version: "encryptionkey@0"
I20260926 21:37:27.231012 6992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:37:27.234468 7007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:27.236053 6992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:27.236259 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal
uuid: "dfe9f0e1bfd544e297a4fe182b445c02"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "ed368d98a9664865739168af95afb16d"
server_key_iv: "80d481396a716ff6d6c9b3a0f2fa1253"
server_key_version: "encryptionkey@0"
I20260926 21:37:27.236493 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:27.252825 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:27.257409 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:27.257674 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:27.270740 6854 heartbeater.cc:503] Master 127.0.97.62:34137 was elected leader, sending a full tablet report...
I20260926 21:37:27.276731 6992 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42509
I20260926 21:37:27.276808 7058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42509 every 8 connection(s)
I20260926 21:37:27.278208 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:37:27.282187 7059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:27.287467 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 6992
I20260926 21:37:27.289239 7059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:37:27 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 21:37:27 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 21:37:27.310925 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.292341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.0.97.62:34137)
0926 21:37:27.292825 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:27.292831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:27.293459 (+ 628us) server_negotiation.cc:407] Connection header received
0926 21:37:27.293807 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:27.293812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:27.293887 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:27.293997 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:27.295708 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.296867 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.299144 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.299764 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.299901 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.304160 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:27.304169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:27.304176 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:27.304223 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:27.307367 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.308313 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.308319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.308323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.308458 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.308928 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.308936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.308938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.309365 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:27.309530 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:27.310323 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 21:37:27.310568 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":259,"threads_started":1}
I20260926 21:37:27.312098 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.292340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:57060)
0926 21:37:27.293295 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:27.293347 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:27.293579 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:27.294082 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:27.294090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:27.294974 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:27.295523 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.295547 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.297014 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.297018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.299002 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.299017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.300077 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.300085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.300283 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.301122 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:27.301153 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:27.303933 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:27.307513 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.307529 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.307544 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.308178 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.308565 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.308568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.308570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.308727 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.309502 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:27.309511 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:27.309963 (+ 452us) client_negotiation.cc:769] Sending connection context
0926 21:37:27.310237 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 21:37:27.310823 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":38,"spinlock_wait_cycles":15616,"thread_start_us":538,"threads_started":1}
I20260926 21:37:27.312114 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.289931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.0.97.61:42509)
0926 21:37:27.290801 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:27.290817 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:27.290866 (+ 49us) server_negotiation.cc:407] Connection header received
0926 21:37:27.293083 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:27.293106 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:27.293591 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:27.293844 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:27.295713 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.297349 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.299064 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.299945 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.300132 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.304905 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:27.304913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:27.304926 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:27.304954 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:27.308010 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.308821 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.308833 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.308837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.308983 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.309448 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.309455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.309459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.309931 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:27.310233 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:27.310591 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 21:37:27.310969 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":643,"thread_start_us":403,"threads_started":1}
I20260926 21:37:27.314157 7059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:37:27 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 21:37:27.326227 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.315029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:43824)
0926 21:37:27.315293 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:27.315314 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:27.315429 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:27.315685 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:27.315688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:27.315999 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:27.316141 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.316148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.317034 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.317037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.317950 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.317960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.318572 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.318576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.318723 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.319240 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:27.319256 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:27.321165 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:27.324113 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.324116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.324119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.324457 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.324775 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.324778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.324780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.324892 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.325478 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:27.325481 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:27.325615 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 21:37:27.325722 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 21:37:27.325932 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 21:37:27.326305 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.314909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.0.97.61:42509)
0926 21:37:27.315080 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:27.315086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:27.315391 (+ 305us) server_negotiation.cc:407] Connection header received
0926 21:37:27.315506 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:27.315510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:27.315574 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:27.315649 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:27.316252 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.316931 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.318066 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.318438 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.318537 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.321430 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:27.321435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:27.321438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:27.321466 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:27.323994 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.324562 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.324566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.324568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.324681 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.325012 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.325016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.325018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.325381 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:27.325494 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:27.325789 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 21:37:27.326052 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":64}
I20260926 21:37:27.331120 7059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Bootstrap starting.
I20260926 21:37:27.334529 7059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:27.335690 7059 log.cc:824] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:27.338596 7059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: No bootstrap required, opened a new log
I20260926 21:37:27.345969 7059 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } }
I20260926 21:37:27.346338 7059 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:27.346443 7059 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe9f0e1bfd544e297a4fe182b445c02, State: Initialized, Role: FOLLOWER
I20260926 21:37:27.347088 7059 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } }
I20260926 21:37:27.347955 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } }
I20260926 21:37:27.348294 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:37:27.349037 7059 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:37:27.358250 7077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:37:27.358443 7077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:37:27.360760 6992 master_runner.cc:428] Detected that this master dfe9f0e1bfd544e297a4fe182b445c02 is joining an existing cluster
I20260926 21:37:27.360854 6992 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:42509
I20260926 21:37:27.371308 7066 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:27.371660 7066 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } }
I20260926 21:37:27.373237 7066 leader_election.cc:288] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcff42acfbd941d28b12e39fd93cd9ce (127.0.97.62:34137)
I20260926 21:37:27.374547 6541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" is_pre_election: true
I20260926 21:37:27.375748 7008 leader_election.cc:302] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: dfe9f0e1bfd544e297a4fe182b445c02; no voters: fcff42acfbd941d28b12e39fd93cd9ce
I20260926 21:37:27.376489 7066 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 21:37:27.383610 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.364917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.0.97.62:34137)
0926 21:37:27.365115 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:27.365121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:27.365930 (+ 809us) server_negotiation.cc:407] Connection header received
0926 21:37:27.366198 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:27.366203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:27.366314 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:27.366401 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:27.368077 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.369301 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.372552 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.373263 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.373372 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.376618 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:27.376626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:27.376632 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:27.376674 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:27.380319 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.381101 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.381107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.381111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.381311 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.381833 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.381839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.381843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.382212 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:27.382332 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:27.382973 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 21:37:27.383216 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:37:27.384735 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.364997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:57076)
0926 21:37:27.365815 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:27.365838 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:27.366042 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:27.366613 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:27.366620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:27.367365 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:27.367871 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.367890 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.369498 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.369506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.372342 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.372357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.373891 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.373900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.374102 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.374988 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:27.375011 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:27.376412 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:27.380469 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.380474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.380478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.380956 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.381454 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.381459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.381462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.381656 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.382426 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:27.382430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:27.382651 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 21:37:27.383869 (+ 1218us) client_negotiation.cc:240] Negotiation successful
0926 21:37:27.384197 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":4,"thread_start_us":4819,"threads_started":1}
I20260926 21:37:27.386159 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.365445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.0.97.61:42509)
0926 21:37:27.365624 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:27.365630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:27.366299 (+ 669us) server_negotiation.cc:407] Connection header received
0926 21:37:27.366507 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:27.366512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:27.366588 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:27.366653 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:27.368116 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.369260 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.376298 (+ 7038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.376989 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.377094 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.379425 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:27.379431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:27.379437 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:27.379471 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:27.383189 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.384025 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.384030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.384033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.384185 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.384613 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.384618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.384621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.384985 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:27.385087 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:27.385634 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 21:37:27.385834 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 21:37:27.387176 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.365065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:43826)
0926 21:37:27.366212 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:27.366230 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:27.366376 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:27.366834 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:27.366839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:27.367616 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:27.367968 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.367980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.369471 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.369476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.374574 (+ 5098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.374594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.377416 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.377424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.377594 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.378157 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:27.378185 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:27.379236 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:27.383424 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.383429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.383433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.383918 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.384310 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.384315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.384318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.384505 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.385140 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:27.385142 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:27.385333 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 21:37:27.386374 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0926 21:37:27.386729 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":2,"thread_start_us":428,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 26 21:37:27 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458645, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 21:37:27.408908 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.391720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.0.97.61:42509)
0926 21:37:27.391984 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:27.391989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:27.393563 (+ 1574us) server_negotiation.cc:407] Connection header received
0926 21:37:27.393883 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:27.393889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:27.393991 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:27.394075 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:27.395575 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.396871 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.398958 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.399701 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.399836 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.403382 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:27.403387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:27.403391 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:27.403424 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:27.406208 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.406896 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.406907 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.406909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.407032 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.407416 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.407421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.407423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.407691 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:27.407841 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:27.408296 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 21:37:27.408521 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260926 21:37:27.408955 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.391720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:43832)
0926 21:37:27.393398 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:27.393447 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:27.393649 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:27.394158 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:27.394168 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:27.395080 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:27.395399 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.395412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.397048 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.397053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.398792 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.398810 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.399889 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.399898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.400145 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.400941 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:27.400970 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:27.403180 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:27.406361 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.406374 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.406382 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.406769 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.407155 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.407160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.407163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.407298 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.407825 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:27.407833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:27.408015 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 21:37:27.408201 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 21:37:27.408530 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1398,"thread_start_us":1233,"threads_started":1}
I20260926 21:37:27.411064 6531 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:42509: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 42509
}
attrs {
promote: true
}
}
dest_uuid: "fcff42acfbd941d28b12e39fd93cd9ce"
cas_config_opid_index: -1
I20260926 21:37:27.412118 6531 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } }
I20260926 21:37:27.417624 7089 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dfe9f0e1bfd544e297a4fe182b445c02 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } } }
I20260926 21:37:27.418890 7034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:27.421263 7089 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:42509
I20260926 21:37:27.421722 7089 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } } }
I20260926 21:37:27.421923 7089 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:27.422092 6992 master.cc:537] Master@127.0.97.61:42509 shutting down...
I20260926 21:37:27.425750 7034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Refusing update from remote peer fcff42acfbd941d28b12e39fd93cd9ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260926 21:37:27.426792 7089 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:27.428323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44739
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34137,127.0.97.61:42509,127.0.97.60:44739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:27.436179 6517 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 21:37:27.436305 6992 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:27.436759 6992 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:27.436995 6992 tablet_replica.cc:330] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: stopping tablet replica
I20260926 21:37:27.444383 6992 master.cc:552] Master@127.0.97.61:42509 shutdown complete.
I20260926 21:37:27.444470 6992 master_runner.cc:305] Clearing existing system tablet
I20260926 21:37:27.444900 6992 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:27.444995 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260926 21:37:27.449066 6517 consensus_peers.cc:487] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce -> Peer dfe9f0e1bfd544e297a4fe182b445c02 (127.0.97.61:42509): Couldn't send request to peer dfe9f0e1bfd544e297a4fe182b445c02. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260926 21:37:27.450054 6992 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 21:37:27.451707 7095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:27.452106 6992 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:27.452527 6992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 21:37:27.452672 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal
uuid: "dfe9f0e1bfd544e297a4fe182b445c02"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "ed368d98a9664865739168af95afb16d"
server_key_iv: "80d481396a716ff6d6c9b3a0f2fa1253"
server_key_version: "encryptionkey@0"
I20260926 21:37:27.457862 6992 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:27.464913 6992 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 21:37:27.465224 6992 log.cc:1196] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 21:37:27.465755 6992 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Deleting consensus metadata
I20260926 21:37:27.466085 6992 master_runner.cc:315] Copying system tablet from 127.0.97.62:34137
I20260926 21:37:27.468571 6992 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:34137
I20260926 21:37:27.490803 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.469427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:57086)
0926 21:37:27.475917 (+ 6490us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:27.475945 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:27.476147 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:27.476586 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:27.476592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:27.477110 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:27.477461 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.477473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.478886 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.478892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.480474 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.480489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.481548 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.481556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.481764 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.482687 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:27.482710 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:27.483790 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:27.487780 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.487786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.487790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.488303 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.488694 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.488701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.488709 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.488860 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.489617 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:27.489622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:27.489825 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 21:37:27.489973 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 21:37:27.490325 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6305,"thread_start_us":6055,"threads_started":1}
I20260926 21:37:27.491851 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.469456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.0.97.62:34137)
0926 21:37:27.469634 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:27.469640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:27.476081 (+ 6441us) server_negotiation.cc:407] Connection header received
0926 21:37:27.476294 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:27.476300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:27.476405 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:27.476520 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:27.477623 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.478739 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.480636 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.481295 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.481436 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.484024 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:27.484031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:27.484036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:27.484074 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:27.487639 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.488432 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.488437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.488440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.488595 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.489000 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.489006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.489009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.489413 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:27.489561 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:27.491263 (+ 1702us) server_negotiation.cc:299] Negotiation successful
0926 21:37:27.491493 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:37:27.493292 6551 tablet_copy_service.cc:140] P fcff42acfbd941d28b12e39fd93cd9ce: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dfe9f0e1bfd544e297a4fe182b445c02 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57086)
I20260926 21:37:27.493566 6551 tablet_copy_service.cc:161] P fcff42acfbd941d28b12e39fd93cd9ce: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dfe9f0e1bfd544e297a4fe182b445c02 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57086: session id = dfe9f0e1bfd544e297a4fe182b445c02-00000000000000000000000000000000
I20260926 21:37:27.498627 6551 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 21:37:27.500965 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 21:37:27.507944 6992 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: tablet copy: Starting download of 0 data blocks...
I20260926 21:37:27.508234 6992 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: tablet copy: Starting download of 1 WAL segments...
I20260926 21:37:27.511049 6992 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 21:37:27.515467 6551 tablet_copy_service.cc:342] P fcff42acfbd941d28b12e39fd93cd9ce: Request end of tablet copy session dfe9f0e1bfd544e297a4fe182b445c02-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57086
I20260926 21:37:27.515641 6551 tablet_copy_service.cc:433] P fcff42acfbd941d28b12e39fd93cd9ce: ending tablet copy session dfe9f0e1bfd544e297a4fe182b445c02-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dfe9f0e1bfd544e297a4fe182b445c02
I20260926 21:37:27.526604 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:27.529496 7102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:27.534548 7103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:27.541457 7105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:27.541687 6992 server_base.cc:1034] running on GCE node
I20260926 21:37:27.542044 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:27.542699 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:27.543928 6992 hybrid_clock.cc:648] HybridClock initialized: now 1790458647543841 us; error 55 us; skew 500 ppm
I20260926 21:37:27.544135 6992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:27 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:27.546465 6992 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 21:37:27.550487 6992 webserver.cc:460] Webserver started at http://127.0.97.61:40009/ using document root <none> and password file <none>
I20260926 21:37:27.550848 6992 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:27.550943 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:27.554059 6992 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 21:37:27.556340 7112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:27.556826 6992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 21:37:27.556964 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal
uuid: "dfe9f0e1bfd544e297a4fe182b445c02"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "ed368d98a9664865739168af95afb16d"
server_key_iv: "80d481396a716ff6d6c9b3a0f2fa1253"
server_key_version: "encryptionkey@0"
I20260926 21:37:27.557126 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:27.592150 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:27.596374 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:27.681169 6992 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42509
I20260926 21:37:27.681100 7163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42509 every 8 connection(s)
I20260926 21:37:27.682523 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:37:27.687985 7164 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:37:27.692726 7164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Bootstrap starting.
I20260926 21:37:27.710119 7164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:37:27.710683 7164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Bootstrap complete.
I20260926 21:37:27.711776 7164 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } }
I20260926 21:37:27.712054 7164 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfe9f0e1bfd544e297a4fe182b445c02, State: Initialized, Role: LEARNER
I20260926 21:37:27.712311 7164 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } }
I20260926 21:37:27.713249 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } } }
I20260926 21:37:27.713562 7164 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:27.713732 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: This master's current role is: LEARNER
I20260926 21:37:27.736526 7178 catalog_manager.cc:1259] Loaded cluster ID: fc41cec6fcd54f518d5ad96e19200613
I20260926 21:37:27.736683 7178 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: loading cluster ID for follower catalog manager: success
I20260926 21:37:27.740334 7178 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: acquiring CA information for follower catalog manager: success
I20260926 21:37:27.743896 7178 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:37:27.784811 6987 heartbeater.cc:503] Master 127.0.97.62:34137 was elected leader, sending a full tablet report...
W20260926 21:37:27.856626 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:27.856896 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:27.857038 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:27.865097 7092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:27.865257 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:27.865340 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:27.865447 7092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:27.865531 7092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:27.875856 7092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34137,127.0.97.61:42509,127.0.97.60:44739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:27.877389 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:27.878680 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:27.891247 7183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:27.891207 7186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:27.891294 7184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:27.894672 7092 server_base.cc:1034] running on GCE node
I20260926 21:37:27.895552 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:27.897002 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:27.898207 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790458647898152 us; error 46 us; skew 500 ppm
I20260926 21:37:27.898582 7092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:27 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:27.901782 7092 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 21:37:27.903215 7092 webserver.cc:460] Webserver started at http://127.0.97.60:36657/ using document root <none> and password file <none>
I20260926 21:37:27.903895 7092 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:27.903995 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:27.904250 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:27.907747 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data/instance:
uuid: "eebddf22663345e78c0d27564976621d"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "2acf3eec1d02b4d26496ecfa45697814"
server_key_iv: "97caab60b52b1f1b6c9f5d11df206f5f"
server_key_version: "encryptionkey@0"
I20260926 21:37:27.908480 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal/instance:
uuid: "eebddf22663345e78c0d27564976621d"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "2acf3eec1d02b4d26496ecfa45697814"
server_key_iv: "97caab60b52b1f1b6c9f5d11df206f5f"
server_key_version: "encryptionkey@0"
I20260926 21:37:27.913816 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 21:37:27.917496 7193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:27.918905 7092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:27.919046 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal
uuid: "eebddf22663345e78c0d27564976621d"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "2acf3eec1d02b4d26496ecfa45697814"
server_key_iv: "97caab60b52b1f1b6c9f5d11df206f5f"
server_key_version: "encryptionkey@0"
I20260926 21:37:27.919229 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:27.936389 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:27.941326 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:27.941579 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:27.961021 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44739
I20260926 21:37:27.961059 7244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44739 every 8 connection(s)
I20260926 21:37:27.962443 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:37:27.966162 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 7092
I20260926 21:37:27.966588 7245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:27.977001 7245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:37:27 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 21:37:27 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:27.997948 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.968711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.0.97.60:44739)
0926 21:37:27.969562 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:27.969578 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:27.969629 (+ 51us) server_negotiation.cc:407] Connection header received
0926 21:37:27.971823 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:27.971858 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:27.972490 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:27.973023 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:27.974502 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.976486 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.977850 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.978839 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.979053 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.985106 (+ 6053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:27.985116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:27.985132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:27.985184 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:27.990629 (+ 5445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.991992 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.992006 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.992020 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.992267 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.992850 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.992858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.992864 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.993798 (+ 934us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:27.995525 (+ 1727us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:27.995893 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 21:37:27.996461 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":404,"threads_started":1}
I20260926 21:37:28.001626 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.980140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.0.97.62:34137)
0926 21:37:27.980389 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:27.980410 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:27.981073 (+ 663us) server_negotiation.cc:407] Connection header received
0926 21:37:27.981334 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:27.981338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:27.981417 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:27.981498 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:27.983658 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.984558 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.986980 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.987760 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.987929 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.993551 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:27.993575 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:27.993583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:27.993637 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:27.998166 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.999112 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.999118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.999121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:27.999275 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:27.999760 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:27.999764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:27.999767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.000123 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.000268 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.001011 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.001265 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:37:28.001655 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.980032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:57096)
0926 21:37:27.980882 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:27.980973 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:27.981168 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:27.981608 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:27.981617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:27.983150 (+ 1533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:27.983481 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.983492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.984810 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.984818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.986797 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:27.986812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:27.988255 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:27.988265 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:27.988571 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:27.989469 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:27.989503 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:27.993289 (+ 3786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:27.998357 (+ 5068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.998365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.998372 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.998962 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:27.999392 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:27.999397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:27.999401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:27.999607 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.000295 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.000306 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.000592 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.000887 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.001236 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":396,"threads_started":1}
I20260926 21:37:28.003051 7245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:37:28.008345 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.987045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:43842)
0926 21:37:27.997377 (+ 10332us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:27.997398 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:27.997527 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.001531 (+ 4004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.001535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.001562 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:28.002034 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.002047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.003660 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.003665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.006219 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.006238 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.007492 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.007498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.007672 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.007736 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.007856 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.008006 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10008,"thread_start_us":10144,"threads_started":1}
I20260926 21:37:28.008414 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:27.987183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.0.97.61:42509)
0926 21:37:28.000999 (+ 13816us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.001005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.001023 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:37:28.001099 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.001104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.001219 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.001469 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:28.002188 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.003484 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.006391 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.007318 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.007469 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.007695 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.007883 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.008021 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13672,"thread_start_us":14139,"threads_started":1}
I20260926 21:37:28.010396 7139 raft_consensus.cc:1203] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 21:37:28.011078 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcff42acfbd941d28b12e39fd93cd9ce. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } } }
I20260926 21:37:28.011346 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: This master's current role is: LEARNER
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 21:37:28.017521 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.003920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.0.97.61:42509)
0926 21:37:28.004521 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.004532 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.004550 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:37:28.004592 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.004595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.004646 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.004760 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.005500 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.006388 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.007675 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.008116 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.008276 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.012312 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.012317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.012321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.012352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.014967 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.015713 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.015717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.015720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.015856 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.016210 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.016213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.016215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.016456 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.016572 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.016943 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.017115 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":393,"threads_started":1}
I20260926 21:37:28.017522 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.003866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:43846)
0926 21:37:28.004074 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.004093 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.004186 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.004783 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.004788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.005171 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.005367 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.005375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.006520 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.006526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.007524 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.007535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.008291 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.008297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.008536 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.009149 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.009166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.012140 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.015151 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.015155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.015158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.015567 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.015965 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.015969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.015971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.016104 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.016557 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.016560 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.016722 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.016860 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.017134 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 21:37:28.018687 7245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458647, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 21:37:28.030988 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.019432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44739 (local address 127.0.0.1:48046)
0926 21:37:28.019644 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.019659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.019748 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.020027 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.020030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.020352 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.020493 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.020500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.021581 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.021585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.022402 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.022412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.023161 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.023165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.023327 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.023962 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.023979 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.025961 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.028892 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.028896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.028898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.029291 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.029678 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.029681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.029683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.029817 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.030293 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.030296 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.030406 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.030511 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.030718 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 21:37:28.031109 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.019423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.0.97.60:44739)
0926 21:37:28.019600 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.019613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.019718 (+ 105us) server_negotiation.cc:407] Connection header received
0926 21:37:28.019833 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.019837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.019919 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.019998 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.020605 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.021466 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.022529 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.023023 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.023119 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.026121 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.026126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.026129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.026155 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.028786 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.029396 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.029407 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.029410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.029560 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.029927 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.029930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.029932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.030200 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.030304 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.030633 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.030842 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":54}
I20260926 21:37:28.035830 7245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: Bootstrap starting.
I20260926 21:37:28.039292 7245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:28.040295 7245 log.cc:824] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:28.043380 7245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: No bootstrap required, opened a new log
I20260926 21:37:28.050282 7245 raft_consensus.cc:348] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } }
I20260926 21:37:28.050647 7245 raft_consensus.cc:374] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:28.050789 7245 raft_consensus.cc:729] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebddf22663345e78c0d27564976621d, State: Initialized, Role: FOLLOWER
I20260926 21:37:28.051391 7245 consensus_queue.cc:260] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } }
I20260926 21:37:28.052445 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } } }
I20260926 21:37:28.052892 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:37:28.053656 7245 sys_catalog.cc:564] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:37:28.063403 7266 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:37:28.063548 7266 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:37:28.067904 7092 master_runner.cc:428] Detected that this master eebddf22663345e78c0d27564976621d is joining an existing cluster
I20260926 21:37:28.068032 7092 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:44739
I20260926 21:37:28.092008 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.072572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.0.97.61:42509)
0926 21:37:28.072757 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.072765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.074422 (+ 1657us) server_negotiation.cc:407] Connection header received
0926 21:37:28.074645 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.074650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.074740 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.074850 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.076645 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.077780 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.079923 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.080587 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.080694 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.084003 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.084009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.084014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.084055 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.088154 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.089012 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.089018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.089022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.089165 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.089693 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.089700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.089706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.090097 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.090218 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.091400 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.091637 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:37:28.093894 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.072381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.0.97.60:44739)
0926 21:37:28.073071 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.073078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.075282 (+ 2204us) server_negotiation.cc:407] Connection header received
0926 21:37:28.075514 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.075526 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.075609 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.075699 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.077540 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.079078 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.081822 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.083196 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.083340 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.085764 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.085770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.085775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.085809 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.089521 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.090296 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.090305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.090308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.090443 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.091165 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.091169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.091172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.091507 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.091699 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.092482 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.093515 (+ 1033us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":850,"server-negotiator.queue_time_us":96}
I20260926 21:37:28.093715 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.073133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44739 (local address 127.0.0.1:48048)
0926 21:37:28.075182 (+ 2049us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.075208 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.075364 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.075924 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.075931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.076915 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.077352 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.077369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.079356 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.079364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.081642 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.081661 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.083406 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.083412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.083607 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.084303 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.084355 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.085574 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.089697 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.089702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.089706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.090190 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.090552 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.090577 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.090580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.091040 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.091749 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.091754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.091952 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.092099 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.093201 (+ 1102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1889,"mutex_wait_us":24,"thread_start_us":1654,"threads_started":1}
I20260926 21:37:28.094286 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.073063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:43852)
0926 21:37:28.074273 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.074298 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.074494 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.075043 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.075052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.075891 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.076454 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.076467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.077983 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.077988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.079719 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.079740 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.081060 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.081067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.081376 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.082323 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.082361 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.083750 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.088345 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.088350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.088354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.088858 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.089338 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.089343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.089347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.089540 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.090317 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.090327 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.091017 (+ 690us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.092284 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.093747 (+ 1463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":773,"thread_start_us":822,"threads_started":1}
I20260926 21:37:28.099545 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.075316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:57108)
0926 21:37:28.083292 (+ 7976us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.083316 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.083511 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.084049 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.084069 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.085861 (+ 1792us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.086271 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.086282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.087958 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.087964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.089991 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.090006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.091249 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.091257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.091449 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.092034 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.092067 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.092997 (+ 930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.096966 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.096969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.096972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.097406 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.097841 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.097844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.097846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.097975 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.098577 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.098580 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.098742 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.098852 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.099196 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7807,"thread_start_us":7624,"threads_started":1}
I20260926 21:37:28.099671 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.072371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.0.97.62:34137)
0926 21:37:28.072595 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.072600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.083409 (+ 10809us) server_negotiation.cc:407] Connection header received
0926 21:37:28.083655 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.083661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.083750 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.083828 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.086473 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.087706 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.090185 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.090864 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.090965 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.093241 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.093248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.093253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.093294 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.096821 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.097548 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.097554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.097558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.097726 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.098094 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.098099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.098103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.098459 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.098626 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.099066 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.099302 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:37:28.104523 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 6592
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458645, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 21:37:28.121317 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.105615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44739 (local address 127.0.0.1:48050)
0926 21:37:28.105799 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.105821 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.105958 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.106781 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.106789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.107274 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.107491 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.107506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.108950 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.108956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.109929 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.109940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.110985 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.110992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.111178 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.111892 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.111916 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.114275 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.118636 (+ 4361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.118643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.118647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.119041 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.119396 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.119402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.119406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.119541 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.120231 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.120237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.120412 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.120584 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.120897 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 21:37:28.121358 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.106303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.0.97.60:44739)
0926 21:37:28.106452 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.106456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.106470 (+ 14us) server_negotiation.cc:407] Connection header received
0926 21:37:28.106542 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.106545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.106630 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.106754 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.107693 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.108813 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.110072 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.110634 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.110760 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.114545 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.114553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.114565 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.114611 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.118438 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.119175 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.119179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.119181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.119282 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.119699 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.119705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.119708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.120088 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.120241 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.120699 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.120945 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 21:37:28.123486 6531 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:44739: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "eebddf22663345e78c0d27564976621d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 44739
}
attrs {
promote: true
}
}
dest_uuid: "fcff42acfbd941d28b12e39fd93cd9ce"
cas_config_opid_index: 5
I20260926 21:37:28.124550 6531 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } }
I20260926 21:37:28.126638 7276 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eebddf22663345e78c0d27564976621d (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.127259 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.105605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.0.97.1:46237)
0926 21:37:28.106341 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.106351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.106381 (+ 30us) server_negotiation.cc:407] Connection header received
0926 21:37:28.106452 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.106461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.106756 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.107009 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.108619 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.110302 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.112094 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.113017 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.113154 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.118798 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.118816 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.118837 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.118890 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.123129 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.124112 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.124122 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.124125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.124285 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.124702 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.124707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.124711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.125148 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.125471 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.126578 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.126854 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":349,"threads_started":1}
I20260926 21:37:28.127426 7139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 LEARNER]: Refusing update from remote peer fcff42acfbd941d28b12e39fd93cd9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 21:37:28.128062 7250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 21:37:28.128798 6715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:28.129552 7220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:28.133144 7220 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 FOLLOWER]: Refusing update from remote peer fcff42acfbd941d28b12e39fd93cd9ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 21:37:28.137132 7276 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:44739
I20260926 21:37:28.140756 7092 master.cc:537] Master@127.0.97.60:44739 shutting down...
I20260926 21:37:28.141683 7276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:28.141870 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.142439 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:28.157158 7139 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eebddf22663345e78c0d27564976621d (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.161710 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: true } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.162009 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: This master's current role is: LEARNER
I20260926 21:37:28.162472 7277 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: attempting to promote NON_VOTER dfe9f0e1bfd544e297a4fe182b445c02 to VOTER
I20260926 21:37:28.163117 7277 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } }
I20260926 21:37:28.165243 7139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 LEARNER]: Refusing update from remote peer fcff42acfbd941d28b12e39fd93cd9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 21:37:28.165848 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 21:37:28.168551 7276 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dfe9f0e1bfd544e297a4fe182b445c02 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.169239 7138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dfe9f0e1bfd544e297a4fe182b445c02 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.172554 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.172922 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:28.173161 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.173549 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:28.174685 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.175001 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:37:28.184003 7092 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:28.184475 7092 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:28.184703 7092 tablet_replica.cc:330] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: stopping tablet replica
W20260926 21:37:28.184798 6516 consensus_peers.cc:487] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce -> Peer eebddf22663345e78c0d27564976621d (127.0.97.60:44739): Couldn't send request to peer eebddf22663345e78c0d27564976621d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 21:37:28.201136 7092 master.cc:552] Master@127.0.97.60:44739 shutdown complete.
I20260926 21:37:28.201287 7092 master_runner.cc:305] Clearing existing system tablet
I20260926 21:37:28.201676 7092 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:28.201797 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:28.204000 7092 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 21:37:28.205400 7284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:28.205718 7092 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:28.205987 7092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 21:37:28.206089 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal
uuid: "eebddf22663345e78c0d27564976621d"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "2acf3eec1d02b4d26496ecfa45697814"
server_key_iv: "97caab60b52b1f1b6c9f5d11df206f5f"
server_key_version: "encryptionkey@0"
I20260926 21:37:28.209465 7092 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:28.214534 7092 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 21:37:28.214694 7092 log.cc:1196] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 21:37:28.215076 7092 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: Deleting consensus metadata
I20260926 21:37:28.215378 7092 master_runner.cc:315] Copying system tablet from 127.0.97.62:34137
I20260926 21:37:28.218701 7092 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:34137
I20260926 21:37:28.233124 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.219464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.0.97.62:34137)
0926 21:37:28.219799 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.219808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.221726 (+ 1918us) server_negotiation.cc:407] Connection header received
0926 21:37:28.221942 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.221946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.222009 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.222119 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.222954 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.223673 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.224969 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.225655 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.225808 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.227714 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.227719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.227723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.227753 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.230485 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.231177 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.231183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.231187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.231318 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.231701 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.231708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.231714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.231977 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.232157 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.232524 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.232737 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260926 21:37:28.233213 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.219355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:57112)
0926 21:37:28.221610 (+ 2255us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.221632 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.221791 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.222140 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.222145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.222594 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.222801 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.222810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.223818 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.223824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.224794 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.224807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.225865 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.225871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.226121 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.226771 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.226788 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.227489 (+ 701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.230645 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.230650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.230654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.231031 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.231436 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.231441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.231445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.231593 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.232113 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.232122 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.232281 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.232423 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.232740 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2114,"thread_start_us":1996,"threads_started":1}
I20260926 21:37:28.234114 6551 tablet_copy_service.cc:140] P fcff42acfbd941d28b12e39fd93cd9ce: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer eebddf22663345e78c0d27564976621d ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57112)
I20260926 21:37:28.234280 6551 tablet_copy_service.cc:161] P fcff42acfbd941d28b12e39fd93cd9ce: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer eebddf22663345e78c0d27564976621d at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57112: session id = eebddf22663345e78c0d27564976621d-00000000000000000000000000000000
I20260926 21:37:28.237255 6551 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 21:37:28.240013 7092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 21:37:28.245127 7092 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: tablet copy: Starting download of 0 data blocks...
I20260926 21:37:28.245476 7092 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: tablet copy: Starting download of 1 WAL segments...
I20260926 21:37:28.248611 7092 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 21:37:28.251791 6551 tablet_copy_service.cc:342] P fcff42acfbd941d28b12e39fd93cd9ce: Request end of tablet copy session eebddf22663345e78c0d27564976621d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57112
I20260926 21:37:28.251935 6551 tablet_copy_service.cc:433] P fcff42acfbd941d28b12e39fd93cd9ce: ending tablet copy session eebddf22663345e78c0d27564976621d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer eebddf22663345e78c0d27564976621d
I20260926 21:37:28.262861 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:28.265508 7291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:28.271492 7292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:28.276149 7294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:28.278458 7092 server_base.cc:1034] running on GCE node
I20260926 21:37:28.278883 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:28.279596 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:28.280797 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790458648280768 us; error 37 us; skew 500 ppm
I20260926 21:37:28.281021 7092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:28.283052 7092 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 21:37:28.284343 7092 webserver.cc:460] Webserver started at http://127.0.97.60:38299/ using document root <none> and password file <none>
I20260926 21:37:28.284693 7092 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:28.284812 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:28.287679 7092 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260926 21:37:28.289455 7301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:28.289845 7092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 21:37:28.289987 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal
uuid: "eebddf22663345e78c0d27564976621d"
format_stamp: "Formatted at 2026-09-26 21:37:27 on dist-test-slave-lhhq"
server_key: "2acf3eec1d02b4d26496ecfa45697814"
server_key_iv: "97caab60b52b1f1b6c9f5d11df206f5f"
server_key_version: "encryptionkey@0"
I20260926 21:37:28.290199 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:28.301627 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:28.304906 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:28.309509 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 6592
I20260926 21:37:28.333259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46237
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34137
--webserver_port=43539
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34137,127.0.97.61:42509
--tserver_master_addrs=127.0.97.62:34137,127.0.97.61:42509,127.0.97.60:44739 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:37:28.351609 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44739
I20260926 21:37:28.351647 7353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44739 every 8 connection(s)
I20260926 21:37:28.352865 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:37:28.358161 7354 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:37:28.362179 7354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: Bootstrap starting.
I20260926 21:37:28.380841 7354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:37:28.381381 7354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: Bootstrap complete.
I20260926 21:37:28.382428 7354 raft_consensus.cc:348] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } }
I20260926 21:37:28.382714 7354 raft_consensus.cc:729] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eebddf22663345e78c0d27564976621d, State: Initialized, Role: LEARNER
I20260926 21:37:28.382958 7354 consensus_queue.cc:260] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } }
I20260926 21:37:28.383782 7357 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.384054 7354 sys_catalog.cc:564] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:28.384186 7357 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: This master's current role is: LEARNER
I20260926 21:37:28.402410 7368 catalog_manager.cc:1259] Loaded cluster ID: fc41cec6fcd54f518d5ad96e19200613
I20260926 21:37:28.402536 7368 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: loading cluster ID for follower catalog manager: success
I20260926 21:37:28.405756 7368 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: acquiring CA information for follower catalog manager: success
I20260926 21:37:28.407984 7368 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:37:28.670122 7369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.662619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44739 (local address 127.0.0.1:35868)
0926 21:37:28.663210 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.663236 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.663430 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.664045 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.664048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.664070 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:28.664257 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.664265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.666181 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.666186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.667442 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.667456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.669477 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.669482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.669633 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.669659 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.669746 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.669852 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":251,"threads_started":1}
I20260926 21:37:28.670251 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.662850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.0.97.60:44739)
0926 21:37:28.663637 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.663645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.663668 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:37:28.663750 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.663757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.663856 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.664002 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:28.664416 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.665997 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.667839 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.669276 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.669435 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.669654 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.669790 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.669924 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":419,"threads_started":1}
I20260926 21:37:28.671120 7328 raft_consensus.cc:1203] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 21:37:28.671716 7357 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcff42acfbd941d28b12e39fd93cd9ce. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:28.672020 7357 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: This master's current role is: LEARNER
W20260926 21:37:28.691807 7334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:28.692108 7334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:28.692270 7334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:28.699995 7334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:28.700150 7334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:28.700318 7334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:28.710168 7334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43539
--tserver_master_addrs=127.0.97.62:34137,127.0.97.61:42509,127.0.97.60:44739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:28.711476 7334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:28.712886 7334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:28.726887 7375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:28.726958 7378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:28.726915 7376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:28.727902 7334 server_base.cc:1034] running on GCE node
I20260926 21:37:28.728602 7334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:28.729713 7334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:28.730958 7334 hybrid_clock.cc:648] HybridClock initialized: now 1790458648730904 us; error 39 us; skew 500 ppm
I20260926 21:37:28.731299 7334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:28.734560 7334 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 21:37:28.736001 7334 webserver.cc:460] Webserver started at http://127.0.97.1:43539/ using document root <none> and password file <none>
I20260926 21:37:28.737133 7334 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:28.737324 7334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:28.742733 7334 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 21:37:28.745621 7385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:28.746945 7334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 21:37:28.747267 7334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "9cab11a3ee124d9093478181be796360"
format_stamp: "Formatted at 2026-09-26 21:37:25 on dist-test-slave-lhhq"
server_key: "adaf504c7f2d1db9c989fc317c6dd8c8"
server_key_iv: "7ee89fae0940e413416ef816294624c1"
server_key_version: "encryptionkey@0"
I20260926 21:37:28.748526 7334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:28.764562 7334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:28.768679 7334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:28.768977 7334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:28.769997 7334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:28.771766 7334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:28.771889 7334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:28.772023 7334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:28.772109 7334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:28.809763 7334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46237
I20260926 21:37:28.809820 7497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46237 every 8 connection(s)
I20260926 21:37:28.811363 7334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:28.816445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 7334
I20260926 21:37:28.816815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 6726
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458648, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 21:37:28 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 21:37:28.846064 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.814522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59687 (local address 127.0.97.62:34137)
0926 21:37:28.817959 (+ 3437us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.817969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.820373 (+ 2404us) server_negotiation.cc:407] Connection header received
0926 21:37:28.823984 (+ 3611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.823990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.824075 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.824241 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.827176 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.828476 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.831827 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.832572 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.832681 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.838253 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.838261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.838266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.838309 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.842306 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.843280 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.843285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.843289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.843431 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.843968 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.843973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.843976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.844325 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.844470 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.845461 (+ 991us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.845685 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3265,"thread_start_us":1196,"threads_started":1}
I20260926 21:37:28.846483 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.814735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33965 (local address 127.0.97.61:42509)
0926 21:37:28.815474 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.815480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.819157 (+ 3677us) server_negotiation.cc:407] Connection header received
0926 21:37:28.825262 (+ 6105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.825269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.825379 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.825467 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.827422 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.828607 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.831838 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.832578 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.832681 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.838837 (+ 6156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.838843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.839296 (+ 453us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.839347 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.843283 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.844119 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.844124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.844128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.844246 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.844666 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.844671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.844674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.845003 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.845130 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.845653 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.845863 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":477,"threads_started":1}
I20260926 21:37:28.847358 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.816808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.1:59687)
0926 21:37:28.820235 (+ 3427us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.820256 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.823500 (+ 3244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.824582 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.824594 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.825681 (+ 1087us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.826917 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.826950 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.828696 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.828701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.831649 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.831659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.833014 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.833023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.833412 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.834632 (+ 1220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.834650 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.837929 (+ 3279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.842480 (+ 4551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.842497 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.842518 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.843135 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.843566 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.843572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.843576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.843837 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.844457 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.844475 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.844959 (+ 484us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.845284 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.845900 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3284,"mutex_wait_us":355,"spinlock_wait_cycles":39040,"thread_start_us":2527,"threads_started":1}
I20260926 21:37:28.847615 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.815923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.97.1:33965)
0926 21:37:28.818959 (+ 3036us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.819028 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.825033 (+ 6005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.825750 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.825755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.826892 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.827215 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.827229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.828842 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.828847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.831637 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.831652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.833069 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.833076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.833421 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.834590 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.834615 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.838535 (+ 3920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.843442 (+ 4907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.843447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.843450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.843993 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.844353 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.844360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.844363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.844541 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.845240 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.845247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.845369 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.846884 (+ 1515us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.847143 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2650,"mutex_wait_us":306,"spinlock_wait_cycles":4352,"thread_start_us":2411,"threads_started":1}
I20260926 21:37:28.848274 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.814741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46723 (local address 127.0.97.60:44739)
0926 21:37:28.815007 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.815017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.819727 (+ 4710us) server_negotiation.cc:407] Connection header received
0926 21:37:28.824300 (+ 4573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.824310 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.824413 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.824497 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.827162 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.828294 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.831805 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.832462 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.832567 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.839617 (+ 7050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.839626 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.839632 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.839678 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.843484 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.845040 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.845046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.845050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.845229 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.846694 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.846701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.846706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.847022 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.847165 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.847701 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.847926 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 21:37:28.849071 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.818458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.0.97.2:36955)
0926 21:37:28.826436 (+ 7978us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:28.826446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:28.826478 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:37:28.826545 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:28.826556 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:28.826857 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:28.827039 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:28.829325 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.831207 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.833165 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.834827 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.834972 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.841108 (+ 6136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:28.841127 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:28.841146 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:28.841208 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:28.845107 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.846004 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.846013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.846016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.846170 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:28.846636 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:28.846643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:28.846648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:28.847025 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:28.847288 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:28.848414 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0926 21:37:28.848673 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7788,"thread_start_us":7611,"threads_started":1}
I20260926 21:37:28.849260 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:28.816122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44739 (local address 127.0.97.1:46723)
0926 21:37:28.819613 (+ 3491us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:28.819630 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:28.824099 (+ 4469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:28.824785 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:28.824793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:28.825836 (+ 1043us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:28.826919 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.826948 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.828521 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.828527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.831631 (+ 3104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:28.831645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:28.832924 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:28.832931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:28.833416 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:28.834313 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:28.834348 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:28.839341 (+ 4993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:28.843657 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.843662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.843666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.844883 (+ 1217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.846408 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:28.846412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:28.846416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:28.846573 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:28.847248 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:28.847252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:28.847399 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 21:37:28.848507 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0926 21:37:28.848759 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3341,"mutex_wait_us":780,"spinlock_wait_cycles":138624,"thread_start_us":3776,"threads_started":1}
I20260926 21:37:28.849319 7500 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42509
I20260926 21:37:28.849642 7500 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:28.850450 7500 heartbeater.cc:511] Master 127.0.97.61:42509 requested a full tablet report, sending...
I20260926 21:37:28.850776 7498 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44739
I20260926 21:37:28.850878 6849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:28.850906 7498 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:28.851436 7498 heartbeater.cc:511] Master 127.0.97.60:44739 requested a full tablet report, sending...
I20260926 21:37:28.851959 7499 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34137
I20260926 21:37:28.852651 7499 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:28.853268 7318 ts_manager.cc:194] Registered new tserver with Master: 9cab11a3ee124d9093478181be796360 (127.0.97.1:46237)
I20260926 21:37:28.853322 7499 heartbeater.cc:511] Master 127.0.97.62:34137 requested a full tablet report, sending...
I20260926 21:37:28.853317 7129 ts_manager.cc:194] Registered new tserver with Master: 9cab11a3ee124d9093478181be796360 (127.0.97.1:46237)
I20260926 21:37:28.854571 6531 ts_manager.cc:194] Re-registered known tserver with Master: 9cab11a3ee124d9093478181be796360 (127.0.97.1:46237)
I20260926 21:37:28.855671 6531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59687
I20260926 21:37:28.938005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 6726
I20260926 21:37:28.954453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36955
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34137
--webserver_port=44869
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34137,127.0.97.61:42509
--tserver_master_addrs=127.0.97.62:34137,127.0.97.61:42509,127.0.97.60:44739 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:37:29.106501 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: true } } }
I20260926 21:37:29.107059 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:29.108618 7511 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: attempting to promote NON_VOTER eebddf22663345e78c0d27564976621d to VOTER
I20260926 21:37:29.109272 7511 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } }
I20260926 21:37:29.111101 7138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Refusing update from remote peer fcff42acfbd941d28b12e39fd93cd9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 21:37:29.111208 7328 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 LEARNER]: Refusing update from remote peer fcff42acfbd941d28b12e39fd93cd9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 21:37:29.111819 7281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 21:37:29.112246 7512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 21:37:29.115319 7138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eebddf22663345e78c0d27564976621d (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:29.117069 7328 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eebddf22663345e78c0d27564976621d (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:29.114620 7281 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eebddf22663345e78c0d27564976621d (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:29.121204 7357 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:29.121541 7357 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:37:29.120998 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:29.121646 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:29.122152 7512 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:29.122450 7512 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:29.122782 7511 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:29.123068 7511 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:29.123435 7515 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:29.123754 7515 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 21:37:29.291270 7510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:29.291535 7510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:29.291693 7510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:29.299037 7510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:29.299154 7510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:29.299350 7510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:29.308954 7510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44869
--tserver_master_addrs=127.0.97.62:34137,127.0.97.61:42509,127.0.97.60:44739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:29.310503 7510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:29.311846 7510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:29.325543 7521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:29.325511 7523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:29.325588 7520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:29.330302 7510 server_base.cc:1034] running on GCE node
I20260926 21:37:29.330956 7510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:29.331949 7510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:29.333173 7510 hybrid_clock.cc:648] HybridClock initialized: now 1790458649333115 us; error 66 us; skew 500 ppm
I20260926 21:37:29.333534 7510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:29 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:29.336678 7510 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 21:37:29.338284 7510 webserver.cc:460] Webserver started at http://127.0.97.2:44869/ using document root <none> and password file <none>
I20260926 21:37:29.339039 7510 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:29.339219 7510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:29.344893 7510 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 21:37:29.347954 7530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:29.349766 7510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:29.349984 7510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "df507c56b90747d89c0d1dd4c88b99cb"
format_stamp: "Formatted at 2026-09-26 21:37:26 on dist-test-slave-lhhq"
server_key: "0352b2a41d4d9fd0d8798836c8cd5221"
server_key_iv: "3daba643fcd5b12e43ef1d0087cc1b01"
server_key_version: "encryptionkey@0"
I20260926 21:37:29.351413 7510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:29.378758 7510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:29.382977 7510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:29.383210 7510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:29.384286 7510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:29.385871 7510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:29.385965 7510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:29.386072 7510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:29.386145 7510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:29.419433 7510 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36955
I20260926 21:37:29.419483 7642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36955 every 8 connection(s)
I20260926 21:37:29.420918 7510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:29.425465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 7510
I20260926 21:37:29.425796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 6859
Sep 26 21:37:29 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 21:37:29 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 21:37:29 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 26 21:37:29 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:29.469900 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:29.424228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37213 (local address 127.0.97.61:42509)
0926 21:37:29.424831 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:29.424837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:29.442640 (+ 17803us) server_negotiation.cc:407] Connection header received
0926 21:37:29.449948 (+ 7308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:29.449957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:29.450053 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:29.450134 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:29.452194 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.453353 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.455269 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.455883 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.456006 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:29.461865 (+ 5859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:29.461875 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:29.461883 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:29.461933 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:29.465694 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:29.466863 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:29.466871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:29.466876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:29.467040 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:29.467683 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:29.467688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:29.467692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:29.468059 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:29.468196 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:29.469210 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0926 21:37:29.469451 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":2146,"threads_started":1}
I20260926 21:37:29.471354 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:29.425293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44739 (local address 127.0.97.2:35515)
0926 21:37:29.446862 (+ 21569us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:29.446879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:29.448379 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:29.449599 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:29.449606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:29.450645 (+ 1039us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:29.451690 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:29.451698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.453222 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.453227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:29.455339 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:29.455353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.456971 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.456977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:29.457167 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:29.457890 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:29.457933 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:29.461108 (+ 3175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:29.466064 (+ 4956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:29.466072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:29.466078 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:29.466765 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:29.467288 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:29.467296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:29.467300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:29.467477 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:29.468210 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:29.468221 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:29.468720 (+ 499us) client_negotiation.cc:769] Sending connection context
0926 21:37:29.469051 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 21:37:29.469691 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21447,"mutex_wait_us":499,"spinlock_wait_cycles":15488,"thread_start_us":21769,"threads_started":1}
I20260926 21:37:29.471738 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:29.423854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54405 (local address 127.0.97.62:34137)
0926 21:37:29.424367 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:29.424374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:29.446382 (+ 22008us) server_negotiation.cc:407] Connection header received
0926 21:37:29.448722 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:29.448729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:29.448813 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:29.448895 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:29.451853 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.453002 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.457323 (+ 4321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.458058 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.458212 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:29.465177 (+ 6965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:29.465208 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:29.465214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:29.465257 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:29.468758 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:29.469592 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:29.469599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:29.469603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:29.469748 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:29.470187 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:29.470192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:29.470196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:29.470525 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:29.470648 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:29.471175 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 21:37:29.471413 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":859,"threads_started":1}
I20260926 21:37:29.472332 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:29.424607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35515 (local address 127.0.97.60:44739)
0926 21:37:29.449267 (+ 24660us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:29.449273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:29.449289 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:37:29.449341 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:29.449349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:29.449424 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:29.449531 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:29.451853 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.453019 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.455511 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.456195 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.456314 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:29.461394 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:29.461399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:29.461403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:29.461434 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:29.465898 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:29.466941 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:29.466949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:29.466957 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:29.467115 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:29.467622 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:29.467627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:29.467630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:29.467967 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:29.468131 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:29.471715 (+ 3584us) server_negotiation.cc:299] Negotiation successful
0926 21:37:29.471956 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24500,"thread_start_us":24343,"threads_started":1}
I20260926 21:37:29.471380 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:29.424497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.97.2:37213)
0926 21:37:29.442425 (+ 17928us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:29.442483 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:29.448069 (+ 5586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:29.451212 (+ 3143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:29.451220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:29.451768 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:29.452019 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:29.452031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.453516 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.453526 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:29.455108 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:29.455119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.456337 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.456348 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:29.457766 (+ 1418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:29.458554 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:29.458657 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:29.461631 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:29.465884 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:29.465897 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:29.465920 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:29.466719 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:29.467202 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:29.467209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:29.467214 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:29.467522 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:29.468299 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:29.468304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:29.468721 (+ 417us) client_negotiation.cc:769] Sending connection context
0926 21:37:29.470445 (+ 1724us) client_negotiation.cc:240] Negotiation successful
0926 21:37:29.470755 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17534,"mutex_wait_us":540,"spinlock_wait_cycles":19200,"thread_start_us":17256,"threads_started":1}
I20260926 21:37:29.472715 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:29.425229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.2:54405)
0926 21:37:29.445796 (+ 20567us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:29.445815 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:29.448562 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:29.449436 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:29.449450 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:29.450815 (+ 1365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:29.451624 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:29.451656 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.453226 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.453230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:29.457151 (+ 3921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:29.457162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.459638 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.459648 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:29.459824 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:29.461622 (+ 1798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:29.461642 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:29.464934 (+ 3292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:29.468918 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:29.468922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:29.468925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:29.469454 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:29.469891 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:29.469897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:29.469900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:29.470053 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:29.470726 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:29.470730 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:29.470882 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 21:37:29.472009 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0926 21:37:29.472302 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20413,"mutex_wait_us":62,"spinlock_wait_cycles":36224,"thread_start_us":22016,"threads_started":1}
I20260926 21:37:29.473001 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:29.426909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.0.97.3:35619)
0926 21:37:29.448311 (+ 21402us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:29.448323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:29.448354 (+ 31us) server_negotiation.cc:407] Connection header received
0926 21:37:29.448429 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:29.448440 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:29.448714 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:29.451263 (+ 2549us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:29.451863 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.453623 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.455445 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:29.456302 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:29.456436 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:29.464044 (+ 7608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:29.464063 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:29.464081 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:29.464131 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:29.468759 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:29.469737 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:29.469746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:29.469749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:29.469920 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:29.470413 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:29.470418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:29.470421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:29.470804 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:29.471121 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:29.472304 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0926 21:37:29.472580 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21217,"thread_start_us":21064,"threads_started":1}
I20260926 21:37:29.474185 7644 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34137
I20260926 21:37:29.474318 7643 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44739
I20260926 21:37:29.474459 6982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:29.474543 7644 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:29.474542 7643 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:29.474185 7645 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42509
I20260926 21:37:29.475234 7645 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:29.475695 7644 heartbeater.cc:511] Master 127.0.97.62:34137 requested a full tablet report, sending...
I20260926 21:37:29.475729 7643 heartbeater.cc:511] Master 127.0.97.60:44739 requested a full tablet report, sending...
I20260926 21:37:29.476476 7645 heartbeater.cc:511] Master 127.0.97.61:42509 requested a full tablet report, sending...
I20260926 21:37:29.477823 6531 ts_manager.cc:194] Re-registered known tserver with Master: df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955)
I20260926 21:37:29.478050 7129 ts_manager.cc:194] Registered new tserver with Master: df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955)
I20260926 21:37:29.478145 7318 ts_manager.cc:194] Registered new tserver with Master: df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955)
I20260926 21:37:29.478960 6531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54405
I20260926 21:37:29.563241 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 6859
I20260926 21:37:29.579310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35619
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34137
--webserver_port=42521
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:34137,127.0.97.61:42509
--tserver_master_addrs=127.0.97.62:34137,127.0.97.61:42509,127.0.97.60:44739 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:37:29.858409 7499 heartbeater.cc:503] Master 127.0.97.62:34137 was elected leader, sending a full tablet report...
W20260926 21:37:29.917717 7656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:29.917999 7656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:29.918092 7656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:29.925345 7656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:29.925484 7656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:29.925662 7656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:29.936208 7656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42521
--tserver_master_addrs=127.0.97.62:34137,127.0.97.61:42509,127.0.97.60:44739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:29.937713 7656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:29.939057 7656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:29.952270 7661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:29.952256 7664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:29.952308 7662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:29.952749 7656 server_base.cc:1034] running on GCE node
I20260926 21:37:29.953656 7656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:29.954634 7656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:29.955858 7656 hybrid_clock.cc:648] HybridClock initialized: now 1790458649955798 us; error 69 us; skew 500 ppm
I20260926 21:37:29.956174 7656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:29 dist-test-slave-lhhq krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:29.959613 7656 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 21:37:29.961050 7656 webserver.cc:460] Webserver started at http://127.0.97.3:42521/ using document root <none> and password file <none>
I20260926 21:37:29.961880 7656 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:29.962037 7656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:29.967489 7656 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:37:29.970259 7671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:29.971695 7656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:29.971827 7656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "a2139015a1b045adabf85d205381c5af"
format_stamp: "Formatted at 2026-09-26 21:37:26 on dist-test-slave-lhhq"
server_key: "ddc9b07b7c04497d37152a176bf97396"
server_key_iv: "83dd8ac481fe637dbb0b9503f0edc361"
server_key_version: "encryptionkey@0"
I20260926 21:37:29.973039 7656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:29.987898 7656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:29.992118 7656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:29.992352 7656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:29.993381 7656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:29.994714 7656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:29.994800 7656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:29.994877 7656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:29.994932 7656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:30.027108 7656 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35619
I20260926 21:37:30.027184 7783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35619 every 8 connection(s)
I20260926 21:37:30.028582 7656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:30.036402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 7656
Sep 26 21:37:30 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 21:37:30 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 21:37:30 dist-test-slave-lhhq krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458649, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 21:37:30.107164 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.031713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55141 (local address 127.0.97.62:34137)
0926 21:37:30.032190 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.032198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.073234 (+ 41036us) server_negotiation.cc:407] Connection header received
0926 21:37:30.073824 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.073835 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.073945 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.074043 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.084806 (+ 10763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.086043 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.089602 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.090224 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.090345 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.096202 (+ 5857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.096208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.096213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.096254 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.100861 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.104194 (+ 3333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.104200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.104203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.104366 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.105030 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.105035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.105038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.105455 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.105611 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.106584 (+ 973us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.106817 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":495,"threads_started":1}
I20260926 21:37:30.108495 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.033870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.3:55141)
0926 21:37:30.073049 (+ 39179us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:30.073079 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:30.073630 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:30.074700 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:30.074720 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:30.077256 (+ 2536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:30.078404 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.078439 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.086586 (+ 8147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.086592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.088781 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.089407 (+ 626us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.091292 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.091300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.092102 (+ 802us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.092840 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:30.092870 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:30.095861 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:30.103369 (+ 7508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:30.103379 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:30.103399 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:30.104071 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:30.104635 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:30.104640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:30.104644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:30.104907 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:30.105639 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:30.105649 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:30.106138 (+ 489us) client_negotiation.cc:769] Sending connection context
0926 21:37:30.106476 (+ 338us) client_negotiation.cc:240] Negotiation successful
0926 21:37:30.107021 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38992,"mutex_wait_us":570,"thread_start_us":39383,"threads_started":1}
I20260926 21:37:30.109516 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.032065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46163 (local address 127.0.97.60:44739)
0926 21:37:30.061681 (+ 29616us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.061704 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.076015 (+ 14311us) server_negotiation.cc:407] Connection header received
0926 21:37:30.078794 (+ 2779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.078804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.078944 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.079048 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.080994 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.082619 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.088842 (+ 6223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.089535 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.089626 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.099116 (+ 9490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.099124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.099130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.099175 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.102898 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.105857 (+ 2959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.105862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.105865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.106006 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.107840 (+ 1834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.107845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.107848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.108200 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.108319 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.108890 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.109134 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29440,"thread_start_us":29294,"threads_started":1}
I20260926 21:37:30.110906 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.031850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39881 (local address 127.0.97.61:42509)
0926 21:37:30.083178 (+ 51328us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.083183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.083198 (+ 15us) server_negotiation.cc:407] Connection header received
0926 21:37:30.083244 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.083248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.083314 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.083444 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.084835 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.086419 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.089800 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.090473 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.090592 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.096970 (+ 6378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.096978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.096984 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.097038 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.103774 (+ 6736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.105318 (+ 1544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.105325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.105328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.105500 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.109177 (+ 3677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.109202 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.109206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.109554 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.109698 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.110325 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.110542 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51209,"thread_start_us":51083,"threads_started":1}
I20260926 21:37:30.110713 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.034071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44739 (local address 127.0.97.3:46163)
0926 21:37:30.075815 (+ 41744us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:30.075849 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:30.076079 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:30.079335 (+ 3256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:30.079343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:30.080111 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:30.080509 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.080525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.083736 (+ 3211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.083745 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.088646 (+ 4901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.088660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.090887 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.090892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.091059 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.091943 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:30.091983 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:30.098849 (+ 6866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:30.104208 (+ 5359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:30.104214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:30.104217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:30.105731 (+ 1514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:30.106148 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:30.106153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:30.106159 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:30.106337 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:30.108406 (+ 2069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:30.108414 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:30.108594 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 21:37:30.110026 (+ 1432us) client_negotiation.cc:240] Negotiation successful
0926 21:37:30.110315 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41430,"mutex_wait_us":9,"thread_start_us":36759,"threads_started":1}
I20260926 21:37:30.111536 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.032693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.97.3:39881)
0926 21:37:30.066477 (+ 33784us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:30.066568 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:30.070169 (+ 3601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:30.083477 (+ 13308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:30.083482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:30.084022 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:30.084309 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.084321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.086555 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.086561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.089280 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.089294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.090705 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.090713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.091048 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.092897 (+ 1849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:30.092933 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:30.096713 (+ 3780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:30.103942 (+ 7229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:30.103948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:30.103954 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:30.104781 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:30.108856 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:30.108861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:30.108864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:30.109037 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:30.109764 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:30.109769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:30.109935 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 21:37:30.110062 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 21:37:30.111255 (+ 1193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33295,"mutex_wait_us":2106,"thread_start_us":47885,"threads_started":1}
I20260926 21:37:30.113991 7785 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34137
I20260926 21:37:30.114177 7786 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42509
I20260926 21:37:30.114380 7786 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:30.114480 7785 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:30.115752 7784 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44739
I20260926 21:37:30.115893 7784 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:30.115986 7785 heartbeater.cc:511] Master 127.0.97.62:34137 requested a full tablet report, sending...
I20260926 21:37:30.116015 7786 heartbeater.cc:511] Master 127.0.97.61:42509 requested a full tablet report, sending...
I20260926 21:37:30.116338 7784 heartbeater.cc:511] Master 127.0.97.60:44739 requested a full tablet report, sending...
I20260926 21:37:30.117981 6531 ts_manager.cc:194] Re-registered known tserver with Master: a2139015a1b045adabf85d205381c5af (127.0.97.3:35619)
I20260926 21:37:30.118206 7129 ts_manager.cc:194] Registered new tserver with Master: a2139015a1b045adabf85d205381c5af (127.0.97.3:35619)
I20260926 21:37:30.118208 7318 ts_manager.cc:194] Registered new tserver with Master: a2139015a1b045adabf85d205381c5af (127.0.97.3:35619)
I20260926 21:37:30.119508 6531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55141
I20260926 21:37:30.137552 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.064790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.0.97.62:34137)
0926 21:37:30.107747 (+ 42957us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.107753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.116917 (+ 9164us) server_negotiation.cc:407] Connection header received
0926 21:37:30.117129 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.117133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.117284 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.117385 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.124007 (+ 6622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.124693 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.126602 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.127016 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.127155 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.131309 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.131316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.131321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.131359 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.134663 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.135355 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.135360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.135362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.135483 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.135867 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.135875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.135877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.136183 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.136436 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.136778 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.137049 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42851,"thread_start_us":47642,"threads_started":1}
I20260926 21:37:30.143455 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.092449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.0.97.60:44739)
0926 21:37:30.109987 (+ 17538us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.109992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.126158 (+ 16166us) server_negotiation.cc:407] Connection header received
0926 21:37:30.126308 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.126315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.126400 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.126518 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.130286 (+ 3768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.131345 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.132751 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.133196 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.133356 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.136246 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.136256 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.136262 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.136296 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.139918 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.141115 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.141122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.141125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.141307 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.141926 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.141932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.141935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.142265 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.142547 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.142869 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.143087 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17434,"thread_start_us":19130,"threads_started":1}
I20260926 21:37:30.142455 6531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:37:30.150868 6531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:37:30.151059 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.065172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.0.97.61:42509)
0926 21:37:30.111122 (+ 45950us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.111131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.117136 (+ 6005us) server_negotiation.cc:407] Connection header received
0926 21:37:30.120366 (+ 3230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.120375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.120479 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.120617 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.126602 (+ 5985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.127344 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.129619 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.130102 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.130238 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.145856 (+ 15618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.145861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.145864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.145899 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.148396 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.149053 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.149058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.149060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.149176 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.149588 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.149593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.149596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.149868 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.150134 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.150428 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.150654 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45762,"thread_start_us":45944,"threads_started":1}
I20260926 21:37:30.183746 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.172938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35619 (local address 127.0.0.1:55706)
0926 21:37:30.176467 (+ 3529us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:30.176484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:30.176641 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:30.177292 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:30.177296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:30.177321 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:30.177689 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.177699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.179760 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.179767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.181241 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.181259 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.182773 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.182779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.182951 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.182983 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 21:37:30.183064 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 21:37:30.183397 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3366,"thread_start_us":2852,"threads_started":1}
I20260926 21:37:30.183846 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.173656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.0.97.3:35619)
0926 21:37:30.174875 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.174885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.176643 (+ 1758us) server_negotiation.cc:407] Connection header received
0926 21:37:30.176806 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.176815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.177080 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.177316 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:30.177849 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.179607 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.181377 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.182334 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.182473 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.183299 (+ 826us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.183372 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.183527 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":889,"threads_started":1}
I20260926 21:37:30.189463 7719 tablet_service.cc:1467] Processing CreateTablet for tablet f3b435352036486db27bfb9411505aba (DEFAULT_TABLE table=test-table [id=3c80e71e856f48069253d75e7e22b13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:30.191025 7719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b435352036486db27bfb9411505aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:30.192387 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.172987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.0.97.1:46237)
0926 21:37:30.176207 (+ 3220us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.176213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.182746 (+ 6533us) server_negotiation.cc:407] Connection header received
0926 21:37:30.182877 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.182889 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.183247 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.183475 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:30.185883 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.187679 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.189989 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.191175 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.191297 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.191537 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.191954 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.192104 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3091,"thread_start_us":3005,"threads_started":1}
I20260926 21:37:30.192436 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.172554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46237 (local address 127.0.0.1:36890)
0926 21:37:30.182393 (+ 9839us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:30.182419 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:30.182576 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:30.184579 (+ 2003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:30.184587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:30.184623 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:30.185687 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.185702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.187843 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.187848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.189798 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.189810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.191660 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.191665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.191776 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.191799 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:37:30.191897 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 21:37:30.192027 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3676,"mutex_wait_us":62,"thread_start_us":2258,"threads_started":1}
I20260926 21:37:30.193468 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.176564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.97.2:36955)
0926 21:37:30.184413 (+ 7849us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.184423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.184449 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:37:30.184538 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.184544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.184724 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.185016 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:30.186654 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.187981 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.190549 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.191864 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.192000 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.192178 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.193025 (+ 847us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.193175 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7655,"thread_start_us":7676,"threads_started":1}
I20260926 21:37:30.193477 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.176144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36955 (local address 127.0.0.1:45026)
0926 21:37:30.184032 (+ 7888us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:30.184049 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:30.184157 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:30.184931 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:30.184936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:30.185001 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:30.185403 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.185479 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.188125 (+ 2646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.188130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.190354 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.190367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.192261 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.192267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.192790 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.192833 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 21:37:30.192920 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 21:37:30.193119 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7770,"mutex_wait_us":285,"thread_start_us":9033,"threads_started":1}
I20260926 21:37:30.196225 7433 tablet_service.cc:1467] Processing CreateTablet for tablet f3b435352036486db27bfb9411505aba (DEFAULT_TABLE table=test-table [id=3c80e71e856f48069253d75e7e22b13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:30.198022 7433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b435352036486db27bfb9411505aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:30.198122 7578 tablet_service.cc:1467] Processing CreateTablet for tablet f3b435352036486db27bfb9411505aba (DEFAULT_TABLE table=test-table [id=3c80e71e856f48069253d75e7e22b13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:30.199352 7578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b435352036486db27bfb9411505aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:30.203328 7820 tablet_bootstrap.cc:492] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: Bootstrap starting.
I20260926 21:37:30.207124 7820 tablet_bootstrap.cc:654] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:30.208048 7820 log.cc:824] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:30.210487 7820 tablet_bootstrap.cc:492] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: No bootstrap required, opened a new log
I20260926 21:37:30.210758 7820 ts_tablet_manager.cc:1397] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 21:37:30.214378 7822 tablet_bootstrap.cc:492] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: Bootstrap starting.
I20260926 21:37:30.215759 7823 tablet_bootstrap.cc:492] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: Bootstrap starting.
I20260926 21:37:30.218039 7822 tablet_bootstrap.cc:654] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:30.218652 7823 tablet_bootstrap.cc:654] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:30.219228 7822 log.cc:824] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:30.219609 7823 log.cc:824] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:30.221468 7820 raft_consensus.cc:348] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } }
I20260926 21:37:30.221827 7820 raft_consensus.cc:374] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:30.221926 7820 raft_consensus.cc:729] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2139015a1b045adabf85d205381c5af, State: Initialized, Role: FOLLOWER
I20260926 21:37:30.222530 7820 consensus_queue.cc:260] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } }
I20260926 21:37:30.222788 7823 tablet_bootstrap.cc:492] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: No bootstrap required, opened a new log
I20260926 21:37:30.222788 7822 tablet_bootstrap.cc:492] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: No bootstrap required, opened a new log
I20260926 21:37:30.223035 7822 ts_tablet_manager.cc:1397] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 21:37:30.223035 7823 ts_tablet_manager.cc:1397] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 21:37:30.224556 7820 ts_tablet_manager.cc:1428] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260926 21:37:30.224813 7785 heartbeater.cc:503] Master 127.0.97.62:34137 was elected leader, sending a full tablet report...
I20260926 21:37:30.234722 7823 raft_consensus.cc:348] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } }
I20260926 21:37:30.235204 7823 raft_consensus.cc:374] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:30.235319 7823 raft_consensus.cc:729] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df507c56b90747d89c0d1dd4c88b99cb, State: Initialized, Role: FOLLOWER
I20260926 21:37:30.236449 7823 consensus_queue.cc:260] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } }
I20260926 21:37:30.237668 7644 heartbeater.cc:503] Master 127.0.97.62:34137 was elected leader, sending a full tablet report...
I20260926 21:37:30.238508 7823 ts_tablet_manager.cc:1428] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 21:37:30.238677 7822 raft_consensus.cc:348] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } }
I20260926 21:37:30.239113 7822 raft_consensus.cc:374] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:30.239212 7822 raft_consensus.cc:729] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cab11a3ee124d9093478181be796360, State: Initialized, Role: FOLLOWER
I20260926 21:37:30.239817 7822 consensus_queue.cc:260] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } }
I20260926 21:37:30.243201 7822 ts_tablet_manager.cc:1428] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: Time spent starting tablet: real 0.020s user 0.013s sys 0.003s
W20260926 21:37:30.309430 7787 tablet.cc:2369] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:37:30.317850 7501 tablet.cc:2369] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:37:30.451002 7646 tablet.cc:2369] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:37:30.588845 7826 raft_consensus.cc:482] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:30.589202 7826 raft_consensus.cc:504] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } }
I20260926 21:37:30.591099 7826 leader_election.cc:288] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cab11a3ee124d9093478181be796360 (127.0.97.1:46237), df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955)
I20260926 21:37:30.596942 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.591570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41607 (local address 127.0.97.2:36955)
0926 21:37:30.591774 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.591780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.592044 (+ 264us) server_negotiation.cc:407] Connection header received
0926 21:37:30.592138 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.592143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.592219 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.592312 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:30.592701 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.593471 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.595212 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.595989 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.596130 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.596225 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.596552 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.596687 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 21:37:30.596961 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.591380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55327 (local address 127.0.97.1:46237)
0926 21:37:30.591576 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.591582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.591712 (+ 130us) server_negotiation.cc:407] Connection header received
0926 21:37:30.591934 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.591942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.592035 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.592136 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:30.592591 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.593462 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.595217 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.596000 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.596148 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.596246 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.596572 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.596704 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 21:37:30.596979 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.591394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36955 (local address 127.0.97.3:41607)
0926 21:37:30.591875 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:30.591894 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:30.592008 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:30.592339 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:30.592342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:30.592361 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:30.592531 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.592539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.593624 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.593632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.595053 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.595065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.596152 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.596158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.596374 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.596422 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 21:37:30.596526 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 21:37:30.596680 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":317}
I20260926 21:37:30.596983 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.591377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46237 (local address 127.0.97.3:55327)
0926 21:37:30.591604 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:30.591626 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:30.591762 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:30.592155 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:30.592159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:30.592183 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:30.592412 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.592436 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.593618 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.593623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.595064 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:30.595075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.596176 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.596181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:30.596414 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.596436 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:37:30.596539 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 21:37:30.596704 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":51}
I20260926 21:37:30.598186 7598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b435352036486db27bfb9411505aba" candidate_uuid: "a2139015a1b045adabf85d205381c5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df507c56b90747d89c0d1dd4c88b99cb" is_pre_election: true
I20260926 21:37:30.598330 7453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b435352036486db27bfb9411505aba" candidate_uuid: "a2139015a1b045adabf85d205381c5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cab11a3ee124d9093478181be796360" is_pre_election: true
I20260926 21:37:30.598701 7598 raft_consensus.cc:2454] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2139015a1b045adabf85d205381c5af in term 0.
I20260926 21:37:30.598817 7453 raft_consensus.cc:2454] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2139015a1b045adabf85d205381c5af in term 0.
I20260926 21:37:30.599386 7674 leader_election.cc:302] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2139015a1b045adabf85d205381c5af, df507c56b90747d89c0d1dd4c88b99cb; no voters:
I20260926 21:37:30.599779 7826 raft_consensus.cc:2781] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:37:30.599921 7826 raft_consensus.cc:482] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:37:30.600047 7826 raft_consensus.cc:3037] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:30.602258 7826 raft_consensus.cc:504] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } }
I20260926 21:37:30.602905 7826 leader_election.cc:288] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [CANDIDATE]: Term 1 election: Requested vote from peers 9cab11a3ee124d9093478181be796360 (127.0.97.1:46237), df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955)
I20260926 21:37:30.603309 7453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b435352036486db27bfb9411505aba" candidate_uuid: "a2139015a1b045adabf85d205381c5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cab11a3ee124d9093478181be796360"
I20260926 21:37:30.603462 7598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b435352036486db27bfb9411505aba" candidate_uuid: "a2139015a1b045adabf85d205381c5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df507c56b90747d89c0d1dd4c88b99cb"
I20260926 21:37:30.603510 7453 raft_consensus.cc:3037] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:30.603694 7598 raft_consensus.cc:3037] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:30.606184 7453 raft_consensus.cc:2454] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2139015a1b045adabf85d205381c5af in term 1.
I20260926 21:37:30.606196 7598 raft_consensus.cc:2454] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2139015a1b045adabf85d205381c5af in term 1.
I20260926 21:37:30.606612 7674 leader_election.cc:302] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9cab11a3ee124d9093478181be796360, a2139015a1b045adabf85d205381c5af, df507c56b90747d89c0d1dd4c88b99cb; no voters:
I20260926 21:37:30.606935 7826 raft_consensus.cc:2781] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:30.607286 7826 raft_consensus.cc:686] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 1 LEADER]: Becoming Leader. State: Replica: a2139015a1b045adabf85d205381c5af, State: Running, Role: LEADER
I20260926 21:37:30.607875 7826 consensus_queue.cc:237] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } }
I20260926 21:37:30.613085 6530 catalog_manager.cc:5495] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af reported cstate change: term changed from 0 to 1, leader changed from <none> to a2139015a1b045adabf85d205381c5af (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a2139015a1b045adabf85d205381c5af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } health_report { overall_health: UNKNOWN } } }
I20260926 21:37:30.645875 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.640660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.0.97.3:35619)
0926 21:37:30.640834 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.640839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.640857 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:37:30.641005 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.641008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.641092 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.641290 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:37:30.641644 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.642313 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.643821 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.644212 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.644299 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.644717 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:37:30.645266 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:37:30.645373 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.645504 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.645567 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:37:30.653681 7453 raft_consensus.cc:1261] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [term 1 FOLLOWER]: Refusing update from remote peer a2139015a1b045adabf85d205381c5af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:30.653690 7598 raft_consensus.cc:1261] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [term 1 FOLLOWER]: Refusing update from remote peer a2139015a1b045adabf85d205381c5af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:30.654659 7829 consensus_queue.cc:1035] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:30.655005 7826 consensus_queue.cc:1035] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [LEADER]: Connected to new peer: Peer: permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:30.671049 7834 mvcc.cc:205] Tried to move back new op lower bound from 7333718633067069440 to 7333718632893468672. Current Snapshot: MvccSnapshot[applied={T|T < 7333718633067069440}]
I20260926 21:37:30.694772 6531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:30.695047 6531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:30.703088 6531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcff42acfbd941d28b12e39fd93cd9ce: Sending DeleteTablet for 3 replicas of tablet f3b435352036486db27bfb9411505aba
I20260926 21:37:30.704244 7433 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b435352036486db27bfb9411505aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36890
I20260926 21:37:30.704603 7719 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b435352036486db27bfb9411505aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55706
I20260926 21:37:30.704591 7578 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b435352036486db27bfb9411505aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45026
I20260926 21:37:30.716992 7842 tablet_replica.cc:330] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: stopping tablet replica
I20260926 21:37:30.717674 7844 tablet_replica.cc:330] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: stopping tablet replica
I20260926 21:37:30.718081 7844 raft_consensus.cc:2229] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:30.718546 7844 raft_consensus.cc:2258] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:30.719291 7843 tablet_replica.cc:330] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: stopping tablet replica
I20260926 21:37:30.719702 7843 raft_consensus.cc:2229] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:37:30.720232 7843 raft_consensus.cc:2258] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:30.720911 7842 raft_consensus.cc:2229] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:30.721560 7842 raft_consensus.cc:2258] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:30.723389 7844 ts_tablet_manager.cc:1905] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:30.727293 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 6499
I20260926 21:37:30.728766 6561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:30.730728 7843 ts_tablet_manager.cc:1905] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:30.735466 7842 ts_tablet_manager.cc:1905] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:30.744843 7843 ts_tablet_manager.cc:1918] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:30.745584 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.707045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.0.97.61:42509)
0926 21:37:30.725682 (+ 18637us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.725688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.726401 (+ 713us) server_negotiation.cc:407] Connection header received
0926 21:37:30.726474 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.726480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.726571 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.726666 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.728986 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.730175 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.732163 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.732784 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.732906 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.736683 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.736691 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.736696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.736740 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.741279 (+ 4539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.742228 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.742235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.742239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.742389 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.742828 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.742833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.742837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.743207 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.743428 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.744816 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.745141 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18496,"thread_start_us":18372,"threads_started":1}
I20260926 21:37:30.748984 7842 ts_tablet_manager.cc:1918] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:30.751075 7844 ts_tablet_manager.cc:1918] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:30.751257 7844 log.cc:1196] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/f3b435352036486db27bfb9411505aba
I20260926 21:37:30.751812 7844 ts_tablet_manager.cc:1939] T f3b435352036486db27bfb9411505aba P df507c56b90747d89c0d1dd4c88b99cb: Deleting consensus metadata
I20260926 21:37:30.752691 7843 log.cc:1196] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/f3b435352036486db27bfb9411505aba
I20260926 21:37:30.749155 7842 log.cc:1196] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/f3b435352036486db27bfb9411505aba
I20260926 21:37:30.753535 7842 ts_tablet_manager.cc:1939] T f3b435352036486db27bfb9411505aba P 9cab11a3ee124d9093478181be796360: Deleting consensus metadata
I20260926 21:37:30.754742 7843 ts_tablet_manager.cc:1939] T f3b435352036486db27bfb9411505aba P a2139015a1b045adabf85d205381c5af: Deleting consensus metadata
I20260926 21:37:30.764009 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.726415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.0.97.60:44739)
0926 21:37:30.734970 (+ 8555us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.734977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.744537 (+ 9560us) server_negotiation.cc:407] Connection header received
0926 21:37:30.744706 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.744714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.744820 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.744954 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.748063 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.749346 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.751423 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.752160 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.752266 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.757127 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.757135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.757141 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.757207 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.760866 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.761763 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.761769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.761773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.761932 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.762387 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.762392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.762396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.762759 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.763020 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.763360 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.763588 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8383,"thread_start_us":9956,"threads_started":1}
I20260926 21:37:30.873139 6518 catalog_manager.cc:4841] TS 9cab11a3ee124d9093478181be796360 (127.0.97.1:46237): tablet f3b435352036486db27bfb9411505aba (table test-table [id=3c80e71e856f48069253d75e7e22b13c]) successfully deleted
I20260926 21:37:30.873093 6517 catalog_manager.cc:4841] TS a2139015a1b045adabf85d205381c5af (127.0.97.3:35619): tablet f3b435352036486db27bfb9411505aba (table test-table [id=3c80e71e856f48069253d75e7e22b13c]) successfully deleted
I20260926 21:37:30.873939 6517 catalog_manager.cc:4841] TS df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955): tablet f3b435352036486db27bfb9411505aba (table test-table [id=3c80e71e856f48069253d75e7e22b13c]) successfully deleted
I20260926 21:37:30.874476 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 6499
I20260926 21:37:30.920917 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.904938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.0.97.60:44739)
0926 21:37:30.905157 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.905163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.905766 (+ 603us) server_negotiation.cc:407] Connection header received
0926 21:37:30.905992 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.905998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.906081 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.906180 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.908391 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.909255 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.910593 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.911021 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.911149 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.915084 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.915097 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.915102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.915136 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.918151 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.918828 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.918834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.918837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.918990 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.919388 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.919393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.919396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.919768 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.920010 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.920357 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.920559 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:37:30.921891 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:30.903685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.0.97.61:42509)
0926 21:37:30.903845 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:30.903850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:30.905636 (+ 1786us) server_negotiation.cc:407] Connection header received
0926 21:37:30.905899 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:30.905908 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:30.906034 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:30.906147 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:30.907765 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.908625 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.910632 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:30.911308 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:30.911439 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:30.915098 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:30.915105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:30.915109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:30.915148 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:30.918907 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.919626 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.919642 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.919645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.919806 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:30.920232 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:30.920237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:30.920241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:30.920554 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:30.920776 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:30.921332 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 21:37:30.921555 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 21:37:31.685530 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:31.684587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.3:54725)
0926 21:37:31.685074 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:31.685265 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":262,"threads_started":1}
W20260926 21:37:31.687305 7785 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34137 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
I20260926 21:37:31.688119 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:31.685658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.2:55679)
0926 21:37:31.687831 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:31.687949 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2065,"thread_start_us":1975,"threads_started":1}
W20260926 21:37:31.688604 7644 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34137 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
I20260926 21:37:31.688640 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:31.686695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.1:35327)
0926 21:37:31.688149 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:31.688281 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1337,"thread_start_us":1206,"threads_started":1}
W20260926 21:37:31.689030 7499 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34137 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
I20260926 21:37:32.202333 7858 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcff42acfbd941d28b12e39fd93cd9ce)
I20260926 21:37:32.202525 7858 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } }
I20260926 21:37:32.203389 7858 leader_election.cc:288] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcff42acfbd941d28b12e39fd93cd9ce (127.0.97.62:34137), eebddf22663345e78c0d27564976621d (127.0.97.60:44739)
I20260926 21:37:32.209601 7859 raft_consensus.cc:482] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcff42acfbd941d28b12e39fd93cd9ce)
I20260926 21:37:32.210237 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.203510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48026)
0926 21:37:32.209809 (+ 6299us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.209976 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6170,"threads_started":1}
I20260926 21:37:32.210146 7859 raft_consensus.cc:504] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } }
W20260926 21:37:32.211228 7113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
I20260926 21:37:32.213439 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.212153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48042)
0926 21:37:32.212794 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.213018 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":368,"threads_started":1}
I20260926 21:37:32.213480 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.212928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48048)
0926 21:37:32.213090 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.213258 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 21:37:32.213719 7113 leader_election.cc:334] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcff42acfbd941d28b12e39fd93cd9ce (127.0.97.62:34137): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
W20260926 21:37:32.213822 7302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
I20260926 21:37:32.215246 7859 leader_election.cc:288] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcff42acfbd941d28b12e39fd93cd9ce (127.0.97.62:34137), dfe9f0e1bfd544e297a4fe182b445c02 (127.0.97.61:42509)
I20260926 21:37:32.216058 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.215720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48054)
0926 21:37:32.215852 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.215931 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 21:37:32.216287 7302 leader_election.cc:334] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcff42acfbd941d28b12e39fd93cd9ce (127.0.97.62:34137): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
I20260926 21:37:32.218681 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.203660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44739 (local address 127.0.0.1:35920)
0926 21:37:32.210451 (+ 6791us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.210470 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:32.210610 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:32.211738 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:32.211743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:32.211768 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:32.212035 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.212049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.214292 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.214298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.216687 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.216698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.217912 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.217918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.218087 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.218111 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 21:37:32.218223 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 21:37:32.218359 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6698,"thread_start_us":6119,"threads_started":1}
I20260926 21:37:32.218736 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.204054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.0.97.60:44739)
0926 21:37:32.211399 (+ 7345us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.211405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.211424 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:37:32.211489 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.211494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.211566 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.212212 (+ 646us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:32.212419 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.214053 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.216869 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.217717 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.217884 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.217989 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.218244 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.218375 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7225,"thread_start_us":7064,"threads_started":1}
I20260926 21:37:32.219411 7328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eebddf22663345e78c0d27564976621d" is_pre_election: true
I20260926 21:37:32.219730 7328 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe9f0e1bfd544e297a4fe182b445c02 in term 1.
I20260926 21:37:32.220194 7114 leader_election.cc:302] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfe9f0e1bfd544e297a4fe182b445c02, eebddf22663345e78c0d27564976621d; no voters: fcff42acfbd941d28b12e39fd93cd9ce
I20260926 21:37:32.220543 7858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 21:37:32.220685 7858 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fcff42acfbd941d28b12e39fd93cd9ce)
I20260926 21:37:32.220784 7858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:37:32.221292 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.212342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:59620)
0926 21:37:32.214479 (+ 2137us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.214502 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:32.214646 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:32.216158 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:32.216162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:32.216196 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:32.216573 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.216592 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.217765 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.217771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.219286 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.219300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.220548 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.220553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.220795 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.220829 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 21:37:32.220921 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 21:37:32.221015 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1974,"thread_start_us":1840,"threads_started":1}
I20260926 21:37:32.221498 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.212928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.0.97.61:42509)
0926 21:37:32.215904 (+ 2976us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.215910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.215928 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:37:32.215977 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.215980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.216030 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.216142 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:32.216768 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.217640 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.219447 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.220377 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.220525 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.220640 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.220957 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.221111 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2825,"thread_start_us":2626,"threads_started":1}
I20260926 21:37:32.222040 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eebddf22663345e78c0d27564976621d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" is_pre_election: true
I20260926 21:37:32.223356 7858 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } }
I20260926 21:37:32.223781 7138 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eebddf22663345e78c0d27564976621d in current term 2: Already voted for candidate dfe9f0e1bfd544e297a4fe182b445c02 in this term.
I20260926 21:37:32.224018 7858 leader_election.cc:288] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [CANDIDATE]: Term 2 election: Requested vote from peers fcff42acfbd941d28b12e39fd93cd9ce (127.0.97.62:34137), eebddf22663345e78c0d27564976621d (127.0.97.60:44739)
I20260926 21:37:32.224481 7328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eebddf22663345e78c0d27564976621d"
I20260926 21:37:32.224588 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.224103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48062)
0926 21:37:32.224334 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.224439 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 21:37:32.224705 7328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:37:32.224781 7304 leader_election.cc:302] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebddf22663345e78c0d27564976621d; no voters: dfe9f0e1bfd544e297a4fe182b445c02, fcff42acfbd941d28b12e39fd93cd9ce
I20260926 21:37:32.225999 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.225690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48078)
0926 21:37:32.225823 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.225896 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 21:37:32.226190 7113 leader_election.cc:334] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcff42acfbd941d28b12e39fd93cd9ce (127.0.97.62:34137): Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
I20260926 21:37:32.227032 7328 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe9f0e1bfd544e297a4fe182b445c02 in term 2.
I20260926 21:37:32.227209 7859 raft_consensus.cc:2726] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 21:37:32.227418 7114 leader_election.cc:302] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfe9f0e1bfd544e297a4fe182b445c02, eebddf22663345e78c0d27564976621d; no voters: fcff42acfbd941d28b12e39fd93cd9ce
I20260926 21:37:32.227703 7858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:37:32.228134 7858 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [term 2 LEADER]: Becoming Leader. State: Replica: dfe9f0e1bfd544e297a4fe182b445c02, State: Running, Role: LEADER
I20260926 21:37:32.228492 7858 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } }
I20260926 21:37:32.230315 7868 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfe9f0e1bfd544e297a4fe182b445c02. Latest consensus state: current_term: 2 leader_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:32.230625 7868 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:32.231206 7870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:32.234663 7870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:32.235471 7870 catalog_manager.cc:1259] Loaded cluster ID: fc41cec6fcd54f518d5ad96e19200613
I20260926 21:37:32.235572 7870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:32.236824 7870 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:32.237749 7870 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Loaded TSK: 0
I20260926 21:37:32.238638 7870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:37:32.256760 7129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:37:32.265161 7129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:37:32.269704 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.269165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48086)
0926 21:37:32.269385 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.269511 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 21:37:32.269753 7328 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [term 2 FOLLOWER]: Refusing update from remote peer dfe9f0e1bfd544e297a4fe182b445c02: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 21:37:32.270494 7868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 21:37:32.270792 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.270413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48092)
0926 21:37:32.270546 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.270615 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 21:37:32.271028 7113 consensus_peers.cc:487] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 -> Peer fcff42acfbd941d28b12e39fd93cd9ce (127.0.97.62:34137): Couldn't send request to peer fcff42acfbd941d28b12e39fd93cd9ce. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 21:37:32.273452 7859 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfe9f0e1bfd544e297a4fe182b445c02. Latest consensus state: current_term: 2 leader_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:32.273736 7859 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:37:32.275308 7868 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:32.275619 7868 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:32.275772 7859 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fcff42acfbd941d28b12e39fd93cd9ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34137 } } peers { permanent_uuid: "dfe9f0e1bfd544e297a4fe182b445c02" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attrs { promote: false } } peers { permanent_uuid: "eebddf22663345e78c0d27564976621d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44739 } attrs { promote: false } } }
I20260926 21:37:32.276005 7859 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebddf22663345e78c0d27564976621d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:37:32.293802 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.284829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.97.2:36955)
0926 21:37:32.285413 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.285423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.285445 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:37:32.285525 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.285533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.285642 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.285769 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:32.286429 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.287874 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.290498 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.292245 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.292407 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.292566 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.293353 (+ 787us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.293508 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":265,"threads_started":1}
I20260926 21:37:32.294489 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.284466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36955 (local address 127.0.0.1:45042)
0926 21:37:32.284597 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.284612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:32.284712 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:32.285819 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:32.285828 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:32.285865 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:32.286249 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.286264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.288034 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.288042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.290268 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.290281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.292827 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.292835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.293126 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.293178 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 21:37:32.293324 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 21:37:32.294014 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":51}
I20260926 21:37:32.295327 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.284026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35619 (local address 127.0.0.1:55722)
0926 21:37:32.284191 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.284213 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:32.284358 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:32.290046 (+ 5688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:32.290056 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:32.290082 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:32.290304 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.290316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.291349 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.291355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.292935 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.292947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.294574 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.294579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.294780 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.294820 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 21:37:32.294912 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 21:37:32.295037 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":26}
I20260926 21:37:32.295358 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.284330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.0.97.3:35619)
0926 21:37:32.289653 (+ 5323us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.289662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.289686 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:37:32.289775 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.289783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.289881 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.290010 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:32.290475 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.291208 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.293079 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.294417 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.294564 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.294787 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.294941 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.295075 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5135,"thread_start_us":5019,"threads_started":1}
I20260926 21:37:32.295553 7578 tablet_service.cc:1467] Processing CreateTablet for tablet 897e3fab9ae04afc91148cb44b9426d5 (DEFAULT_TABLE table=test-table [id=9eac474e42364fa6b84cff6ff2ba5fb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:32.296033 7578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e3fab9ae04afc91148cb44b9426d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:32.296016 7719 tablet_service.cc:1467] Processing CreateTablet for tablet 897e3fab9ae04afc91148cb44b9426d5 (DEFAULT_TABLE table=test-table [id=9eac474e42364fa6b84cff6ff2ba5fb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:32.296419 7719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e3fab9ae04afc91148cb44b9426d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:32.301821 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.285076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.0.97.1:46237)
0926 21:37:32.285626 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.285635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.295720 (+ 10085us) server_negotiation.cc:407] Connection header received
0926 21:37:32.295797 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.295802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.295876 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.295980 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:32.296361 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.297388 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.299539 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.300756 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.300882 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.300977 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.301310 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.301447 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":277,"threads_started":1}
I20260926 21:37:32.301821 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.284953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46237 (local address 127.0.0.1:36896)
0926 21:37:32.295354 (+ 10401us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.295373 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:32.295594 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:32.296023 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:32.296029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:32.296058 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:32.296195 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.296204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.297551 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.297558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.299371 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.299384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.300918 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.300924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.301130 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.301161 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 21:37:32.301274 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 21:37:32.301407 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10311,"threads_started":1}
I20260926 21:37:32.305694 7882 tablet_bootstrap.cc:492] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: Bootstrap starting.
I20260926 21:37:32.306710 7881 tablet_bootstrap.cc:492] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: Bootstrap starting.
I20260926 21:37:32.307459 7882 tablet_bootstrap.cc:654] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:32.307796 7433 tablet_service.cc:1467] Processing CreateTablet for tablet 897e3fab9ae04afc91148cb44b9426d5 (DEFAULT_TABLE table=test-table [id=9eac474e42364fa6b84cff6ff2ba5fb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:32.308240 7433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e3fab9ae04afc91148cb44b9426d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:32.308353 7881 tablet_bootstrap.cc:654] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:32.309415 7882 tablet_bootstrap.cc:492] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: No bootstrap required, opened a new log
I20260926 21:37:32.309557 7882 ts_tablet_manager.cc:1397] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260926 21:37:32.310501 7882 raft_consensus.cc:348] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } }
I20260926 21:37:32.310747 7882 raft_consensus.cc:374] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:32.310880 7882 raft_consensus.cc:729] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df507c56b90747d89c0d1dd4c88b99cb, State: Initialized, Role: FOLLOWER
I20260926 21:37:32.311033 7881 tablet_bootstrap.cc:492] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: No bootstrap required, opened a new log
I20260926 21:37:32.311200 7881 ts_tablet_manager.cc:1397] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 21:37:32.311242 7882 consensus_queue.cc:260] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } }
I20260926 21:37:32.312062 7881 raft_consensus.cc:348] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } }
I20260926 21:37:32.312376 7881 raft_consensus.cc:374] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:32.312491 7881 raft_consensus.cc:729] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2139015a1b045adabf85d205381c5af, State: Initialized, Role: FOLLOWER
I20260926 21:37:32.312759 7881 consensus_queue.cc:260] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } }
I20260926 21:37:32.315094 7887 tablet_bootstrap.cc:492] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: Bootstrap starting.
I20260926 21:37:32.316152 7882 ts_tablet_manager.cc:1428] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260926 21:37:32.321892 7645 heartbeater.cc:503] Master 127.0.97.61:42509 was elected leader, sending a full tablet report...
I20260926 21:37:32.322615 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.318941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.3:46545)
0926 21:37:32.322117 (+ 3176us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.322280 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2998,"thread_start_us":2860,"threads_started":1}
I20260926 21:37:32.323079 7887 tablet_bootstrap.cc:654] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:32.323410 7881 ts_tablet_manager.cc:1428] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20260926 21:37:32.323777 7786 heartbeater.cc:503] Master 127.0.97.61:42509 was elected leader, sending a full tablet report...
I20260926 21:37:32.324944 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.323995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.3:46783)
0926 21:37:32.324641 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.324766 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506}
I20260926 21:37:32.330003 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.316632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.2:39549)
0926 21:37:32.329680 (+ 13048us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.329790 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12922,"thread_start_us":12777,"threads_started":1}
I20260926 21:37:32.330214 7887 tablet_bootstrap.cc:492] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: No bootstrap required, opened a new log
I20260926 21:37:32.330382 7887 ts_tablet_manager.cc:1397] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260926 21:37:32.330974 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.330652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.2:42415)
0926 21:37:32.330791 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.330867 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 21:37:32.331058 7887 raft_consensus.cc:348] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } }
I20260926 21:37:32.331301 7887 raft_consensus.cc:374] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:32.331413 7887 raft_consensus.cc:729] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cab11a3ee124d9093478181be796360, State: Initialized, Role: FOLLOWER
I20260926 21:37:32.331670 7887 consensus_queue.cc:260] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } }
I20260926 21:37:32.332401 7887 ts_tablet_manager.cc:1428] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 21:37:32.333581 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.332633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.1:44251)
0926 21:37:32.333158 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.333328 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":269,"threads_started":1}
I20260926 21:37:32.334569 7500 heartbeater.cc:503] Master 127.0.97.61:42509 was elected leader, sending a full tablet report...
I20260926 21:37:32.335397 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.334181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.1:37681)
0926 21:37:32.335168 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.335247 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":764,"threads_started":1}
I20260926 21:37:32.619458 7891 raft_consensus.cc:482] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:32.619726 7891 raft_consensus.cc:504] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } }
I20260926 21:37:32.621436 7891 leader_election.cc:288] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2139015a1b045adabf85d205381c5af (127.0.97.3:35619), df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955)
I20260926 21:37:32.627198 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.621863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59667 (local address 127.0.97.3:35619)
0926 21:37:32.622088 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.622096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.622120 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:37:32.622195 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.622200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.622280 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.622411 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:32.622943 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.623644 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.625497 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.626261 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.626404 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.626525 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.626798 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.626926 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 21:37:32.627240 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.622213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36317 (local address 127.0.97.2:36955)
0926 21:37:32.622430 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.622437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.622458 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:37:32.622534 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.622538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.622590 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.622699 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:32.623129 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.623793 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.625546 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.626299 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.626436 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.626544 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.626851 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.626979 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:37:32.627286 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.621702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35619 (local address 127.0.97.1:59667)
0926 21:37:32.621899 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.621920 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:32.622054 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:32.622438 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:32.622443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:32.622469 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:32.622786 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.622794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.623772 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.623780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.625341 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.625355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.626444 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.626449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.626633 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.626656 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:37:32.626760 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 21:37:32.626911 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":112}
I20260926 21:37:32.627734 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.622054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36955 (local address 127.0.97.1:36317)
0926 21:37:32.622289 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.622306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:32.622405 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:32.622718 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:32.622722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:32.622746 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:32.622996 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.623003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.623933 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.623939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.625395 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:32.625406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.626532 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.626538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:32.626699 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.626721 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:37:32.626825 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 21:37:32.626986 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":25}
I20260926 21:37:32.628109 7739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e3fab9ae04afc91148cb44b9426d5" candidate_uuid: "9cab11a3ee124d9093478181be796360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2139015a1b045adabf85d205381c5af" is_pre_election: true
I20260926 21:37:32.628371 7598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e3fab9ae04afc91148cb44b9426d5" candidate_uuid: "9cab11a3ee124d9093478181be796360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df507c56b90747d89c0d1dd4c88b99cb" is_pre_election: true
I20260926 21:37:32.628541 7739 raft_consensus.cc:2454] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cab11a3ee124d9093478181be796360 in term 0.
I20260926 21:37:32.628620 7598 raft_consensus.cc:2454] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cab11a3ee124d9093478181be796360 in term 0.
I20260926 21:37:32.629060 7388 leader_election.cc:302] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cab11a3ee124d9093478181be796360, a2139015a1b045adabf85d205381c5af; no voters:
I20260926 21:37:32.629441 7891 raft_consensus.cc:2781] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:37:32.629568 7891 raft_consensus.cc:482] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:37:32.629669 7891 raft_consensus.cc:3037] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:32.631748 7891 raft_consensus.cc:504] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } }
I20260926 21:37:32.632299 7891 leader_election.cc:288] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [CANDIDATE]: Term 1 election: Requested vote from peers a2139015a1b045adabf85d205381c5af (127.0.97.3:35619), df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955)
I20260926 21:37:32.632768 7739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e3fab9ae04afc91148cb44b9426d5" candidate_uuid: "9cab11a3ee124d9093478181be796360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2139015a1b045adabf85d205381c5af"
I20260926 21:37:32.632917 7598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e3fab9ae04afc91148cb44b9426d5" candidate_uuid: "9cab11a3ee124d9093478181be796360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df507c56b90747d89c0d1dd4c88b99cb"
I20260926 21:37:32.633116 7598 raft_consensus.cc:3037] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:32.633103 7739 raft_consensus.cc:3037] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:32.635217 7598 raft_consensus.cc:2454] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cab11a3ee124d9093478181be796360 in term 1.
I20260926 21:37:32.635268 7739 raft_consensus.cc:2454] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cab11a3ee124d9093478181be796360 in term 1.
I20260926 21:37:32.635664 7388 leader_election.cc:302] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cab11a3ee124d9093478181be796360, a2139015a1b045adabf85d205381c5af; no voters:
I20260926 21:37:32.635982 7891 raft_consensus.cc:2781] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:32.636415 7891 raft_consensus.cc:686] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 1 LEADER]: Becoming Leader. State: Replica: 9cab11a3ee124d9093478181be796360, State: Running, Role: LEADER
I20260926 21:37:32.636838 7891 consensus_queue.cc:237] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } }
I20260926 21:37:32.637552 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.636996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.1:46407)
0926 21:37:32.637207 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.637298 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 21:37:32.640849 7127 catalog_manager.cc:5495] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cab11a3ee124d9093478181be796360 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9cab11a3ee124d9093478181be796360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cab11a3ee124d9093478181be796360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46237 } health_report { overall_health: HEALTHY } } }
I20260926 21:37:32.680440 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.674878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.0.97.1:46237)
0926 21:37:32.675043 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.675047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.675064 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:37:32.675222 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.675226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.675299 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.675455 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:37:32.675824 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.676507 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.678035 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.678704 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.678824 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.679172 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:37:32.679890 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:37:32.680058 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.680102 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.680182 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:37:32.684706 7598 raft_consensus.cc:1261] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [term 1 FOLLOWER]: Refusing update from remote peer 9cab11a3ee124d9093478181be796360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:32.685132 7739 raft_consensus.cc:1261] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [term 1 FOLLOWER]: Refusing update from remote peer 9cab11a3ee124d9093478181be796360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:32.685482 7894 consensus_queue.cc:1035] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df507c56b90747d89c0d1dd4c88b99cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:32.685858 7891 consensus_queue.cc:1035] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2139015a1b045adabf85d205381c5af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:32.691005 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.688362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.3:49423)
0926 21:37:32.690361 (+ 1999us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.690469 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1876}
I20260926 21:37:32.694248 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.688258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.2:41773)
0926 21:37:32.693979 (+ 5721us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.694083 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5572}
I20260926 21:37:32.701555 7897 mvcc.cc:205] Tried to move back new op lower bound from 7333718641389776896 to 7333718641206202368. Current Snapshot: MvccSnapshot[applied={T|T < 7333718641389776896}]
I20260926 21:37:32.704088 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.702606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.3:45109)
0926 21:37:32.703820 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.703915 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1099}
I20260926 21:37:32.704577 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.703988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.1:56745)
0926 21:37:32.704295 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.704385 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260926 21:37:32.706486 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.706072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.1:42997)
0926 21:37:32.706259 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.706333 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 21:37:32.709934 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.709456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.97.2:44397)
0926 21:37:32.709673 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.709765 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 21:37:32.716949 7127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:32.717257 7127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:32.723389 7127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfe9f0e1bfd544e297a4fe182b445c02: Sending DeleteTablet for 3 replicas of tablet 897e3fab9ae04afc91148cb44b9426d5
I20260926 21:37:32.724416 7719 tablet_service.cc:1514] Processing DeleteTablet for tablet 897e3fab9ae04afc91148cb44b9426d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:32 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55722
I20260926 21:37:32.724666 7578 tablet_service.cc:1514] Processing DeleteTablet for tablet 897e3fab9ae04afc91148cb44b9426d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:32 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:45042
I20260926 21:37:32.725517 7906 tablet_replica.cc:330] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: stopping tablet replica
I20260926 21:37:32.725802 7906 raft_consensus.cc:2229] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:32.725817 7433 tablet_service.cc:1514] Processing DeleteTablet for tablet 897e3fab9ae04afc91148cb44b9426d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:32 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:36896
I20260926 21:37:32.726123 7907 tablet_replica.cc:330] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: stopping tablet replica
I20260926 21:37:32.726307 7906 raft_consensus.cc:2258] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:32.726341 7907 raft_consensus.cc:2229] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:32.726681 7907 raft_consensus.cc:2258] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:32.727344 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 7334
I20260926 21:37:32.727777 7907 ts_tablet_manager.cc:1905] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:32.730494 7906 ts_tablet_manager.cc:1905] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:32.735502 7907 ts_tablet_manager.cc:1918] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:32.735649 7907 log.cc:1196] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/897e3fab9ae04afc91148cb44b9426d5
I20260926 21:37:32.736123 7907 ts_tablet_manager.cc:1939] T 897e3fab9ae04afc91148cb44b9426d5 P df507c56b90747d89c0d1dd4c88b99cb: Deleting consensus metadata
I20260926 21:37:32.737461 7115 catalog_manager.cc:4841] TS df507c56b90747d89c0d1dd4c88b99cb (127.0.97.2:36955): tablet 897e3fab9ae04afc91148cb44b9426d5 (table test-table [id=9eac474e42364fa6b84cff6ff2ba5fb7]) successfully deleted
I20260926 21:37:32.738086 7908 tablet_replica.cc:330] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: stopping tablet replica
I20260926 21:37:32.738299 7908 raft_consensus.cc:2229] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:37:32.738927 7908 raft_consensus.cc:2258] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:32.739678 7906 ts_tablet_manager.cc:1918] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:32.739847 7906 log.cc:1196] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/897e3fab9ae04afc91148cb44b9426d5
I20260926 21:37:32.740011 7908 ts_tablet_manager.cc:1905] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:32.740329 7906 ts_tablet_manager.cc:1939] T 897e3fab9ae04afc91148cb44b9426d5 P a2139015a1b045adabf85d205381c5af: Deleting consensus metadata
I20260926 21:37:32.741523 7115 catalog_manager.cc:4841] TS a2139015a1b045adabf85d205381c5af (127.0.97.3:35619): tablet 897e3fab9ae04afc91148cb44b9426d5 (table test-table [id=9eac474e42364fa6b84cff6ff2ba5fb7]) successfully deleted
I20260926 21:37:32.747041 7908 ts_tablet_manager.cc:1918] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:32.747197 7908 log.cc:1196] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/897e3fab9ae04afc91148cb44b9426d5
I20260926 21:37:32.747778 7908 ts_tablet_manager.cc:1939] T 897e3fab9ae04afc91148cb44b9426d5 P 9cab11a3ee124d9093478181be796360: Deleting consensus metadata
I20260926 21:37:32.748953 7116 catalog_manager.cc:4841] TS 9cab11a3ee124d9093478181be796360 (127.0.97.1:46237): tablet 897e3fab9ae04afc91148cb44b9426d5 (table test-table [id=9eac474e42364fa6b84cff6ff2ba5fb7]) successfully deleted
I20260926 21:37:32.753741 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.729438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.0.97.1:46237)
0926 21:37:32.729621 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.729630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.739030 (+ 9400us) server_negotiation.cc:407] Connection header received
0926 21:37:32.739557 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.739562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.739661 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.739808 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:32.741939 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.742799 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.744015 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.744465 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.744610 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.747147 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:32.747163 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:32.747190 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:32.747231 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:32.750586 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:32.751308 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:32.751323 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:32.751326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:32.751478 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:32.751944 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:32.751949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:32.751952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:32.752467 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:32.752779 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.753011 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.753307 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:37:32.754492 7493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:32.855239 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 7334
I20260926 21:37:32.859153 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.858379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48100)
0926 21:37:32.858883 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.858982 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260926 21:37:32.860383 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.860047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34137 (local address 127.0.0.1:48102)
0926 21:37:32.860195 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:32.860282 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 21:37:32.874420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 7510
I20260926 21:37:32.888833 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:32.875147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.0.97.2:36955)
0926 21:37:32.875414 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:32.875421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:32.875568 (+ 147us) server_negotiation.cc:407] Connection header received
0926 21:37:32.875798 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:32.875807 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:32.875880 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:32.876013 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:32.877609 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.878492 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.879875 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:32.880292 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:32.880430 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:32.883184 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:32.883200 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:32.883217 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:32.883247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:32.886143 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:32.886828 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:32.886836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:32.886838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:32.886971 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:32.887344 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:32.887350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:32.887353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:32.887688 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:32.887990 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:32.888281 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 21:37:32.888497 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 21:37:32.889621 7638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:32.982565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 7510
I20260926 21:37:33.000905 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 7656
I20260926 21:37:33.014709 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:33.001826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.0.97.3:35619)
0926 21:37:33.002007 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:33.002012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:33.002031 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:37:33.002225 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:33.002229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:33.002325 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:33.002420 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:33.003749 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:33.004480 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:33.005773 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:33.006166 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:33.006271 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:33.008761 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:33.008775 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:33.008792 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:33.008837 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:33.011875 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:33.012562 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:33.012570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:33.012572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:33.012721 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:33.013076 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:33.013080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:33.013082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:33.013567 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:33.013816 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:33.014134 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 21:37:33.014343 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:37:33.015408 7779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:33.134441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 7656
I20260926 21:37:33.157018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 6992
I20260926 21:37:33.158332 7159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:33.281509 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 6992
W20260926 21:37:33.302996 7303 connection.cc:535] server connection from 127.0.0.1:35920 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 21:37:33.303369 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 7092
I20260926 21:37:33.304364 7349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:33.420469 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 7092
2026-09-26T21:37:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10036 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:37:33 dist-test-slave-lhhq krb5kdc[7919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:37:33 dist-test-slave-lhhq krb5kdc[7919](info): set up 2 sockets
Sep 26 21:37:33 dist-test-slave-lhhq krb5kdc[7919](info): commencing operation
krb5kdc: starting...
W20260926 21:37:34.522200 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:37:34 dist-test-slave-lhhq krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 21:37:34.737979 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35409
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:35.061336 7932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:35.061702 7932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:35.061858 7932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:35.069365 7932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:35.069507 7932 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 21:37:35.069610 7932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:35.069725 7932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:35.069823 7932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:35.069914 7932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:35.080009 7932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35409
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:35.081453 7932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:35.082808 7932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:35.094849 7938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:35.094870 7940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:35.094928 7937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:35.095654 7932 server_base.cc:1034] running on GCE node
I20260926 21:37:35.096475 7932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:35.097941 7932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:35.099207 7932 hybrid_clock.cc:648] HybridClock initialized: now 1790458655099126 us; error 57 us; skew 500 ppm
I20260926 21:37:35.099573 7932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:35 dist-test-slave-lhhq krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458655, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:35.102662 7932 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 21:37:35.104041 7932 webserver.cc:460] Webserver started at http://127.0.97.62:43807/ using document root <none> and password file <none>
I20260926 21:37:35.104755 7932 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:35.104897 7932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:35.105285 7932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:35.108258 7932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "932af892f0f5478ab4ef158e8eaea7b7"
format_stamp: "Formatted at 2026-09-26 21:37:35 on dist-test-slave-lhhq"
server_key: "8b22e662b7b5613f2dcd8fb1201329d5"
server_key_iv: "0bbdb762fa72de7c307ae5cadf6a5a15"
server_key_version: "encryptionkey@0"
I20260926 21:37:35.109086 7932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "932af892f0f5478ab4ef158e8eaea7b7"
format_stamp: "Formatted at 2026-09-26 21:37:35 on dist-test-slave-lhhq"
server_key: "8b22e662b7b5613f2dcd8fb1201329d5"
server_key_iv: "0bbdb762fa72de7c307ae5cadf6a5a15"
server_key_version: "encryptionkey@0"
I20260926 21:37:35.114342 7932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:37:35.117810 7947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:35.119119 7932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:35.119306 7932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "932af892f0f5478ab4ef158e8eaea7b7"
format_stamp: "Formatted at 2026-09-26 21:37:35 on dist-test-slave-lhhq"
server_key: "8b22e662b7b5613f2dcd8fb1201329d5"
server_key_iv: "0bbdb762fa72de7c307ae5cadf6a5a15"
server_key_version: "encryptionkey@0"
I20260926 21:37:35.119527 7932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:35.137807 7932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:35.141721 7932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:35.142002 7932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:35.163281 7932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35409
I20260926 21:37:35.163293 8008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35409 every 8 connection(s)
I20260926 21:37:35.164660 7932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:35.168502 8009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:35.168999 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 7932
I20260926 21:37:35.169257 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:35.170033 388 external_mini_cluster.cc:1428] Setting key a108cc489d9f4b1507e7a59b0a3903ff
I20260926 21:37:35.180843 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7: Bootstrap starting.
Sep 26 21:37:35 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:35.184240 8009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:35.185325 8009 log.cc:824] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:35.188060 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7: No bootstrap required, opened a new log
I20260926 21:37:35.191900 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:35.172147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.0.97.62:35409)
0926 21:37:35.172923 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:35.172945 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:35.172986 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:37:35.174987 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:35.175008 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:35.175421 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:35.175820 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:35.177258 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.178562 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.179816 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.180488 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.180682 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:35.184846 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:35.184854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:35.184869 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:35.184909 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:35.187968 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:35.188662 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:35.188668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:35.188674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:35.188825 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:35.189243 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:35.189248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:35.189250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:35.189774 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:35.190069 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:35.190545 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 21:37:35.190935 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":328,"threads_started":1}
I20260926 21:37:35.196265 8009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932af892f0f5478ab4ef158e8eaea7b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35409 } }
I20260926 21:37:35.196640 8009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:35.196753 8009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932af892f0f5478ab4ef158e8eaea7b7, State: Initialized, Role: FOLLOWER
I20260926 21:37:35.197374 8009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932af892f0f5478ab4ef158e8eaea7b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35409 } }
I20260926 21:37:35.197616 8009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:35.197767 8009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:35.197935 8009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:35.200316 8009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932af892f0f5478ab4ef158e8eaea7b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35409 } }
I20260926 21:37:35.200805 8009 leader_election.cc:302] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932af892f0f5478ab4ef158e8eaea7b7; no voters:
I20260926 21:37:35.201390 8009 leader_election.cc:288] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:35.201587 8014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:35.202898 8014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [term 1 LEADER]: Becoming Leader. State: Replica: 932af892f0f5478ab4ef158e8eaea7b7, State: Running, Role: LEADER
I20260926 21:37:35.203373 8014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932af892f0f5478ab4ef158e8eaea7b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35409 } }
I20260926 21:37:35.203586 8009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:35.206247 8016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932af892f0f5478ab4ef158e8eaea7b7. Latest consensus state: current_term: 1 leader_uuid: "932af892f0f5478ab4ef158e8eaea7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932af892f0f5478ab4ef158e8eaea7b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35409 } } }
I20260926 21:37:35.206670 8016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:35.207150 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932af892f0f5478ab4ef158e8eaea7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932af892f0f5478ab4ef158e8eaea7b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35409 } } }
I20260926 21:37:35.207443 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7 [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:35.207672 8023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:35.213498 8023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:35.222263 8023 catalog_manager.cc:1347] Generated new cluster ID: 0920700f42e04f2692b757b06ebda309
I20260926 21:37:35.222409 8023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:35.230777 8023 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:35.232023 8023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:35.245265 8023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7: Generated new TSK 0
I20260926 21:37:35.247300 8023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 21:37:35.313076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35409
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:35.643093 8035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:35.643388 8035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:35.643575 8035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:35.650799 8035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:35.650909 8035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:35.651062 8035 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:37:35.651185 8035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:35.660866 8035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:35.662240 8035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:35.663643 8035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:35.676862 8040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:35.676937 8041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:35.676898 8043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:35.678294 8035 server_base.cc:1034] running on GCE node
I20260926 21:37:35.678980 8035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:35.679908 8035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:35.681253 8035 hybrid_clock.cc:648] HybridClock initialized: now 1790458655681140 us; error 48 us; skew 500 ppm
I20260926 21:37:35.681589 8035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:35 dist-test-slave-lhhq krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458655, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:35.684950 8035 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 21:37:35.686483 8035 webserver.cc:460] Webserver started at http://127.0.97.1:36359/ using document root <none> and password file <none>
I20260926 21:37:35.687211 8035 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:35.687337 8035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:35.687666 8035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:35.690623 8035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "987167ed599c4851974443caa40d7224"
format_stamp: "Formatted at 2026-09-26 21:37:35 on dist-test-slave-lhhq"
server_key: "b75195bdc0616ef4ee1c263597317e7e"
server_key_iv: "a1d09a4f23fa63d481d2e118845534b9"
server_key_version: "encryptionkey@0"
I20260926 21:37:35.691398 8035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "987167ed599c4851974443caa40d7224"
format_stamp: "Formatted at 2026-09-26 21:37:35 on dist-test-slave-lhhq"
server_key: "b75195bdc0616ef4ee1c263597317e7e"
server_key_iv: "a1d09a4f23fa63d481d2e118845534b9"
server_key_version: "encryptionkey@0"
I20260926 21:37:35.696354 8035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:37:35.699813 8050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:35.701133 8035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:37:35.701524 8035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "987167ed599c4851974443caa40d7224"
format_stamp: "Formatted at 2026-09-26 21:37:35 on dist-test-slave-lhhq"
server_key: "b75195bdc0616ef4ee1c263597317e7e"
server_key_iv: "a1d09a4f23fa63d481d2e118845534b9"
server_key_version: "encryptionkey@0"
I20260926 21:37:35.701788 8035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:35.733369 8035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:35.737788 8035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:35.738077 8035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:35.740701 8035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:35.740839 8035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:35.741014 8035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:35.741108 8035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:37:35 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458655, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:35.768339 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:35.742570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35409 (local address 127.0.97.1:44803)
0926 21:37:35.743528 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:35.743594 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:35.746556 (+ 2962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:35.747894 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:35.747909 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:35.748922 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:35.750086 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:35.750126 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.751585 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.751589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:35.753352 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:35.753375 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.754393 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.754403 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:35.754797 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:35.755782 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:35.755805 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:35.759510 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:35.763327 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:35.763337 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:35.763356 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:35.764081 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:35.764673 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:35.764683 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:35.764687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:35.764955 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:35.765726 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:35.765734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:35.766349 (+ 615us) client_negotiation.cc:769] Sending connection context
0926 21:37:35.766695 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 21:37:35.767230 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":365,"threads_started":1}
I20260926 21:37:35.769590 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:35.741536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44803 (local address 127.0.97.62:35409)
0926 21:37:35.742003 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:35.742010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:35.743802 (+ 1792us) server_negotiation.cc:407] Connection header received
0926 21:37:35.746991 (+ 3189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:35.746999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:35.747104 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:35.747225 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:35.750344 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.751391 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.753521 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.754082 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.754248 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:35.759870 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:35.759875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:35.759879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:35.759913 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:35.763162 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:35.764326 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:35.764333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:35.764336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:35.764511 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:35.765114 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:35.765119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:35.765128 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:35.765561 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:35.765732 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:35.768871 (+ 3139us) server_negotiation.cc:299] Negotiation successful
0926 21:37:35.769122 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":1324,"threads_started":1}
I20260926 21:37:35.791422 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:35.776756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49777 (local address 127.0.97.62:35409)
0926 21:37:35.776997 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:35.777006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:35.777682 (+ 676us) server_negotiation.cc:407] Connection header received
0926 21:37:35.777988 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:35.777994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:35.778084 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:35.778190 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:35.779125 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.779855 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.781176 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.781741 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.781882 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:35.784413 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:35.784418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:35.784421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:35.784452 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:35.787809 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:35.788796 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:35.788804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:35.788811 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:35.788977 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:35.789484 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:35.789489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:35.789493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:35.789911 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:35.790089 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:35.790650 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 21:37:35.790940 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:37:35.791687 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:35.776773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35409 (local address 127.0.97.1:49777)
0926 21:37:35.777546 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:35.777568 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:35.777817 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:35.778218 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:35.778223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:35.778696 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:35.778973 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:35.778982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.779984 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.780025 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:35.781031 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:35.781043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:35.781976 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:35.781985 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:35.782275 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:35.783107 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:35.783123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:35.784217 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:35.788076 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:35.788083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:35.788091 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:35.788636 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:35.789114 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:35.789120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:35.789127 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:35.789334 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:35.790055 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:35.790062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:35.790314 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 21:37:35.790533 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 21:37:35.791023 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":466,"threads_started":1}
I20260926 21:37:35.793723 8035 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36801
I20260926 21:37:35.793804 8170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36801 every 8 connection(s)
I20260926 21:37:35.795027 8035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:35.797327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 8035
I20260926 21:37:35.797435 8171 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35409
I20260926 21:37:35.797560 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:37:35.797695 8171 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:35.798367 388 external_mini_cluster.cc:1428] Setting key 9d7bbf97ea4b44dec4360c1fbd1b5454
I20260926 21:37:35.798503 8171 heartbeater.cc:511] Master 127.0.97.62:35409 requested a full tablet report, sending...
I20260926 21:37:35.800609 7963 ts_manager.cc:194] Registered new tserver with Master: 987167ed599c4851974443caa40d7224 (127.0.97.1:36801)
I20260926 21:37:35.802126 7963 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44803
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 21:37:35.847015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35409
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:36.169373 8175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:36.169662 8175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:36.169837 8175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:36.177248 8175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:36.177357 8175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:36.177529 8175 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:37:36.177661 8175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:36.187386 8175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:36.188673 8175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:36.190222 8175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:36.203514 8180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:36.203475 8181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:36.203436 8183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:36.204416 8175 server_base.cc:1034] running on GCE node
I20260926 21:37:36.205143 8175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:36.206254 8175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:36.207515 8175 hybrid_clock.cc:648] HybridClock initialized: now 1790458656207415 us; error 73 us; skew 500 ppm
I20260926 21:37:36.207803 8175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:36 dist-test-slave-lhhq krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458656, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:36.211126 8175 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 21:37:36.212663 8175 webserver.cc:460] Webserver started at http://127.0.97.2:37961/ using document root <none> and password file <none>
I20260926 21:37:36.213419 8175 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:36.213511 8175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:36.213793 8175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:36.216729 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "c985c57f3f374b39adc4e704aaa782a8"
format_stamp: "Formatted at 2026-09-26 21:37:36 on dist-test-slave-lhhq"
server_key: "c9e4ef99c743f790af38661aafadc6e0"
server_key_iv: "23ecb9ca6bc9547051ba30986c5850c1"
server_key_version: "encryptionkey@0"
I20260926 21:37:36.217577 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c985c57f3f374b39adc4e704aaa782a8"
format_stamp: "Formatted at 2026-09-26 21:37:36 on dist-test-slave-lhhq"
server_key: "c9e4ef99c743f790af38661aafadc6e0"
server_key_iv: "23ecb9ca6bc9547051ba30986c5850c1"
server_key_version: "encryptionkey@0"
I20260926 21:37:36.222728 8175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:37:36.226284 8190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:36.227766 8175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:37:36.227900 8175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "c985c57f3f374b39adc4e704aaa782a8"
format_stamp: "Formatted at 2026-09-26 21:37:36 on dist-test-slave-lhhq"
server_key: "c9e4ef99c743f790af38661aafadc6e0"
server_key_iv: "23ecb9ca6bc9547051ba30986c5850c1"
server_key_version: "encryptionkey@0"
I20260926 21:37:36.228107 8175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:36.245496 8175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:36.250051 8175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:36.250283 8175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:36.252945 8175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:36.253057 8175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:36.253224 8175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:36.253293 8175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 26 21:37:36 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458656, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:36.278343 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.253393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33625 (local address 127.0.97.62:35409)
0926 21:37:36.253554 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:36.253559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:36.255011 (+ 1452us) server_negotiation.cc:407] Connection header received
0926 21:37:36.256971 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:36.256976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:36.257071 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:36.257179 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:36.259352 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.260389 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.263722 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.264116 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.264295 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.269293 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:36.269298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:36.269302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:36.269335 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:36.273421 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.275223 (+ 1802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.275232 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.275239 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.275454 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.276127 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.276135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.276141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.276490 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:36.276652 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:36.277756 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0926 21:37:36.277982 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:37:36.279479 8199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.254073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35409 (local address 127.0.97.2:33625)
0926 21:37:36.254846 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:36.254888 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:36.256684 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:36.257467 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:36.257484 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:36.258247 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:36.259177 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.259195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.260613 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.260620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.263573 (+ 2953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.263587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.264410 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.264419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.264859 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.265833 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:36.265876 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:36.268982 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:36.273752 (+ 4770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:36.273769 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:36.273829 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:36.274987 (+ 1158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:36.275618 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:36.275624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:36.275627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:36.275935 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:36.276626 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:36.276635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:36.277024 (+ 389us) client_negotiation.cc:769] Sending connection context
0926 21:37:36.277387 (+ 363us) client_negotiation.cc:240] Negotiation successful
0926 21:37:36.278155 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":338,"threads_started":1}
I20260926 21:37:36.307054 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.292437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54289 (local address 127.0.97.62:35409)
0926 21:37:36.292624 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:36.292630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:36.292821 (+ 191us) server_negotiation.cc:407] Connection header received
0926 21:37:36.293094 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:36.293100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:36.293260 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:36.293352 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:36.295063 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.296213 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.298290 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.298722 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.298861 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.300618 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:36.300624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:36.300629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:36.300670 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:36.303738 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.304636 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.304641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.304645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.304782 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.305278 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.305286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.305290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.305832 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:36.306001 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:36.306466 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 21:37:36.306702 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:37:36.307271 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.291941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35409 (local address 127.0.97.2:54289)
0926 21:37:36.292693 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:36.292733 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:36.292938 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:36.293379 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:36.293387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:36.294423 (+ 1036us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:36.294881 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.294898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.296418 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.296427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.298139 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.298153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.298909 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.298915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.299069 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.299711 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:36.299739 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:36.300433 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:36.303871 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:36.303876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:36.303883 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:36.304517 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:36.304893 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:36.304897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:36.304901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:36.305043 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:36.305974 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:36.305981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:36.306181 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 21:37:36.306379 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 21:37:36.306758 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":684,"threads_started":1}
I20260926 21:37:36.311859 8175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39219
I20260926 21:37:36.311877 8309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39219 every 8 connection(s)
I20260926 21:37:36.313006 8175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:36.315414 8310 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35409
I20260926 21:37:36.315646 8310 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:36.316270 8310 heartbeater.cc:511] Master 127.0.97.62:35409 requested a full tablet report, sending...
I20260926 21:37:36.317523 7963 ts_manager.cc:194] Registered new tserver with Master: c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219)
I20260926 21:37:36.318425 7963 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33625
I20260926 21:37:36.318627 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 8175
I20260926 21:37:36.318847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:37:36.319360 388 external_mini_cluster.cc:1428] Setting key e3cec5b3ed69ddba85124c308587ecca
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 21:37:36.366905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35409
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:36.701942 8314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:36.702282 8314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:36.702453 8314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:36.710119 8314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:36.710286 8314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:36.710428 8314 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 21:37:36.710561 8314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:36.720412 8314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:36.721884 8314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:36.723358 8314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:36.737624 8322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:36.737601 8320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:36.737679 8319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:36.738353 8314 server_base.cc:1034] running on GCE node
I20260926 21:37:36.739019 8314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:36.739982 8314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:36.741287 8314 hybrid_clock.cc:648] HybridClock initialized: now 1790458656741141 us; error 59 us; skew 500 ppm
I20260926 21:37:36.741582 8314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:36 dist-test-slave-lhhq krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458656, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:36.744640 8314 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 21:37:36.746320 8314 webserver.cc:460] Webserver started at http://127.0.97.3:38515/ using document root <none> and password file <none>
I20260926 21:37:36.747011 8314 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:36.747140 8314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:36.747466 8314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:36.750399 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "ca564b7ce4b046999d826c988f873132"
format_stamp: "Formatted at 2026-09-26 21:37:36 on dist-test-slave-lhhq"
server_key: "61893f76bca5b23f41de0e7ab945a370"
server_key_iv: "867bb02fde8a04af895a276cd4896d22"
server_key_version: "encryptionkey@0"
I20260926 21:37:36.751199 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ca564b7ce4b046999d826c988f873132"
format_stamp: "Formatted at 2026-09-26 21:37:36 on dist-test-slave-lhhq"
server_key: "61893f76bca5b23f41de0e7ab945a370"
server_key_iv: "867bb02fde8a04af895a276cd4896d22"
server_key_version: "encryptionkey@0"
I20260926 21:37:36.756286 8314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260926 21:37:36.759682 8329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:36.760963 8314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 21:37:36.761132 8314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "ca564b7ce4b046999d826c988f873132"
format_stamp: "Formatted at 2026-09-26 21:37:36 on dist-test-slave-lhhq"
server_key: "61893f76bca5b23f41de0e7ab945a370"
server_key_iv: "867bb02fde8a04af895a276cd4896d22"
server_key_version: "encryptionkey@0"
I20260926 21:37:36.761456 8314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:36.786631 8314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:36.791028 8314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:36.791342 8314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:36.794312 8314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:36.794443 8314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:36.794612 8314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:36.794692 8314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:36.805166 8171 heartbeater.cc:503] Master 127.0.97.62:35409 was elected leader, sending a full tablet report...
Sep 26 21:37:36 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458656, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:36.823258 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.795177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34715 (local address 127.0.97.62:35409)
0926 21:37:36.795413 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:36.795423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:36.797080 (+ 1657us) server_negotiation.cc:407] Connection header received
0926 21:37:36.800264 (+ 3184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:36.800269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:36.800351 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:36.800462 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:36.803332 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.804380 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.808136 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.808573 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.808733 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.815154 (+ 6421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:36.815160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:36.815164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:36.815201 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:36.818139 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.819444 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.819453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.819458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.819649 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.820374 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.820382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.820386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.820827 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:36.821005 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:36.822450 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0926 21:37:36.822761 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:37:36.826467 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.795758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35409 (local address 127.0.97.3:34715)
0926 21:37:36.796855 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:36.796922 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:36.799853 (+ 2931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:36.800793 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:36.800808 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:36.801781 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:36.803010 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.803050 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.804577 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.804585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.807915 (+ 3330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.807940 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.808863 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.808879 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.809332 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.810405 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:36.810447 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:36.814788 (+ 4341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:36.818333 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:36.818351 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:36.818379 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:36.819263 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:36.819816 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:36.819828 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:36.819835 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:36.820215 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:36.820996 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:36.821011 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:36.821783 (+ 772us) client_negotiation.cc:769] Sending connection context
0926 21:37:36.823500 (+ 1717us) client_negotiation.cc:240] Negotiation successful
0926 21:37:36.824287 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":438,"threads_started":1}
I20260926 21:37:36.844542 8314 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41235
I20260926 21:37:36.844558 8448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41235 every 8 connection(s)
I20260926 21:37:36.846208 8314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:36.848379 8449 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35409
I20260926 21:37:36.849102 8449 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:36.849776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 8314
I20260926 21:37:36.849977 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:37:36.850005 8449 heartbeater.cc:511] Master 127.0.97.62:35409 requested a full tablet report, sending...
I20260926 21:37:36.850500 388 external_mini_cluster.cc:1428] Setting key 4ba3155c968f98156bf42450936f895a
I20260926 21:37:36.851572 7963 ts_manager.cc:194] Registered new tserver with Master: ca564b7ce4b046999d826c988f873132 (127.0.97.3:41235)
I20260926 21:37:36.851635 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.834845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35409 (local address 127.0.97.3:50645)
0926 21:37:36.835514 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:36.835533 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:36.835667 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:36.836263 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:36.836271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:36.837011 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:36.837621 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.837638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.839234 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.839242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.841641 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.841659 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.842664 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.842673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.842844 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.843811 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:36.843839 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:36.845109 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:36.848755 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:36.848760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:36.848763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:36.849359 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:36.849742 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:36.849749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:36.849752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:36.849939 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:36.850507 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:36.850511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:36.850720 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 21:37:36.850858 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 21:37:36.851191 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":382,"threads_started":1}
I20260926 21:37:36.852615 7963 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34715
I20260926 21:37:36.851609 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.834825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50645 (local address 127.0.97.62:35409)
0926 21:37:36.835037 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:36.835044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:36.835638 (+ 594us) server_negotiation.cc:407] Connection header received
0926 21:37:36.835790 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:36.835798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:36.835914 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:36.836028 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:36.837818 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.839051 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.841807 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.842467 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.842570 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.845358 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:36.845363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:36.845366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:36.845398 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:36.848645 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.849490 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.849502 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.849504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.849631 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.850096 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.850101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.850104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.850379 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:36.850526 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:36.850981 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 21:37:36.851193 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:37:36.853678 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 21:37:36 dist-test-slave-lhhq krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 21:37:36 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:36 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 21:37:36.903095 8059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:36.892945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.0.97.62:35409)
0926 21:37:36.893197 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:36.893204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:36.893496 (+ 292us) server_negotiation.cc:407] Connection header received
0926 21:37:36.893644 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:36.893648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:36.893712 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:36.893799 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:36.895172 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.895923 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.897326 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.897784 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.897894 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.902880 (+ 4986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45626: BlockingRecv error: recv got EOF from 127.0.0.1:45626 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 21:37:36 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:36.918929 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.906234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.0.97.62:35409)
0926 21:37:36.906480 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:36.906486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:36.906794 (+ 308us) server_negotiation.cc:407] Connection header received
0926 21:37:36.907019 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:36.907023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:36.907108 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:36.907191 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:36.908276 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.908977 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.910252 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.910666 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.910759 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.914231 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:36.914236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:36.914239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:36.914271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:36.916612 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.917321 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.917325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.917327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.917445 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:36.917765 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:36.917768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:36.917770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:36.918013 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:36.918215 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:36.918434 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 21:37:36.918608 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:37:36.923170 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:37:36.930431 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:37:36.956153 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.942123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41235 (local address 127.0.0.1:55786)
0926 21:37:36.943583 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:36.943630 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:36.943835 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:36.945054 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:36.945065 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:36.945107 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:36.945701 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.945715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.948403 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.948411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.951664 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.951684 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.954680 (+ 2996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.954689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.955002 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.955073 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 21:37:36.955271 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 21:37:36.955544 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"spinlock_wait_cycles":4992,"thread_start_us":1274,"threads_started":1}
I20260926 21:37:36.957568 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.942249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.0.97.3:41235)
0926 21:37:36.942979 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:36.942988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:36.944020 (+ 1032us) server_negotiation.cc:407] Connection header received
0926 21:37:36.944122 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:36.944130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:36.944418 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:36.944645 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:36.945992 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.948172 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.951911 (+ 3739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.954012 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.954177 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.954506 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:36.956832 (+ 2326us) server_negotiation.cc:299] Negotiation successful
0926 21:37:36.957082 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":392,"threads_started":1}
I20260926 21:37:36.962428 8379 tablet_service.cc:1467] Processing CreateTablet for tablet 3eae0bf12ddc4bff939558bbaa6176e0 (DEFAULT_TABLE table=test-table [id=52779608ab8a42d283171597d999b98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:36.963747 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eae0bf12ddc4bff939558bbaa6176e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:36.965246 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.942822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.0.97.2:39219)
0926 21:37:36.958292 (+ 15470us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:36.958305 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:36.958343 (+ 38us) server_negotiation.cc:407] Connection header received
0926 21:37:36.958475 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:36.958488 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:36.958784 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:36.959013 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:36.959453 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.960889 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.963086 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.964260 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.964399 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.964598 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:36.964791 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 21:37:36.964935 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15253,"thread_start_us":15103,"threads_started":1}
I20260926 21:37:36.965301 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.942265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39219 (local address 127.0.0.1:58958)
0926 21:37:36.943676 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:36.943692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:36.943835 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:36.958998 (+ 15163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:36.959002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:36.959027 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:36.959305 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.959314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.960993 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.960999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.962911 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.962924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.964456 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.964461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.964615 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.964650 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:37:36.964744 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 21:37:36.964874 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"mutex_wait_us":4,"spinlock_wait_cycles":2688,"thread_start_us":872,"threads_started":1}
I20260926 21:37:36.966826 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.944001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36801 (local address 127.0.0.1:34874)
0926 21:37:36.956475 (+ 12474us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:36.956501 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:36.956631 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:36.959556 (+ 2925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:36.959562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:36.959589 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:36.959808 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.959819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.961750 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.961755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.963584 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:36.963596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.965509 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.965514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:36.965713 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.965755 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 21:37:36.966391 (+ 636us) client_negotiation.cc:240] Negotiation successful
0926 21:37:36.966519 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12339,"mutex_wait_us":1,"thread_start_us":14408,"threads_started":1}
I20260926 21:37:36.967414 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:36.944296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.0.97.1:36801)
0926 21:37:36.958946 (+ 14650us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:36.958954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:36.958981 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:37:36.959085 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:36.959092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:36.959314 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:36.959547 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:36.960134 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.961596 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.963748 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:36.965176 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:36.965319 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:36.966246 (+ 927us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:36.967065 (+ 819us) server_negotiation.cc:299] Negotiation successful
0926 21:37:36.967223 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14495,"thread_start_us":14372,"threads_started":1}
I20260926 21:37:36.969134 8240 tablet_service.cc:1467] Processing CreateTablet for tablet 3eae0bf12ddc4bff939558bbaa6176e0 (DEFAULT_TABLE table=test-table [id=52779608ab8a42d283171597d999b98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:36.970645 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eae0bf12ddc4bff939558bbaa6176e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:36.971304 8101 tablet_service.cc:1467] Processing CreateTablet for tablet 3eae0bf12ddc4bff939558bbaa6176e0 (DEFAULT_TABLE table=test-table [id=52779608ab8a42d283171597d999b98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:36.972434 8101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eae0bf12ddc4bff939558bbaa6176e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:36.978734 8473 tablet_bootstrap.cc:492] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: Bootstrap starting.
I20260926 21:37:36.982491 8473 tablet_bootstrap.cc:654] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:36.983842 8473 log.cc:824] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:36.984272 8474 tablet_bootstrap.cc:492] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: Bootstrap starting.
I20260926 21:37:36.985970 8476 tablet_bootstrap.cc:492] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: Bootstrap starting.
I20260926 21:37:36.987802 8474 tablet_bootstrap.cc:654] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:36.987850 8473 tablet_bootstrap.cc:492] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: No bootstrap required, opened a new log
I20260926 21:37:36.988109 8473 ts_tablet_manager.cc:1397] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260926 21:37:36.988757 8476 tablet_bootstrap.cc:654] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:36.989111 8474 log.cc:824] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:36.989903 8476 log.cc:824] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:36.992182 8474 tablet_bootstrap.cc:492] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: No bootstrap required, opened a new log
I20260926 21:37:36.992527 8474 ts_tablet_manager.cc:1397] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260926 21:37:36.995244 8476 tablet_bootstrap.cc:492] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: No bootstrap required, opened a new log
I20260926 21:37:36.995492 8476 ts_tablet_manager.cc:1397] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 21:37:36.998461 8473 raft_consensus.cc:348] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } }
I20260926 21:37:36.998935 8473 raft_consensus.cc:374] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:36.999028 8473 raft_consensus.cc:729] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca564b7ce4b046999d826c988f873132, State: Initialized, Role: FOLLOWER
I20260926 21:37:36.999666 8473 consensus_queue.cc:260] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } }
I20260926 21:37:37.000824 8449 heartbeater.cc:503] Master 127.0.97.62:35409 was elected leader, sending a full tablet report...
I20260926 21:37:37.001575 8473 ts_tablet_manager.cc:1428] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260926 21:37:37.003513 8474 raft_consensus.cc:348] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } }
I20260926 21:37:37.003945 8474 raft_consensus.cc:374] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:37.003796 8476 raft_consensus.cc:348] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } }
I20260926 21:37:37.004076 8474 raft_consensus.cc:729] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c985c57f3f374b39adc4e704aaa782a8, State: Initialized, Role: FOLLOWER
I20260926 21:37:37.004238 8476 raft_consensus.cc:374] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:37.004341 8476 raft_consensus.cc:729] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987167ed599c4851974443caa40d7224, State: Initialized, Role: FOLLOWER
I20260926 21:37:37.004779 8474 consensus_queue.cc:260] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } }
I20260926 21:37:37.004905 8476 consensus_queue.cc:260] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } }
I20260926 21:37:37.009153 8310 heartbeater.cc:503] Master 127.0.97.62:35409 was elected leader, sending a full tablet report...
I20260926 21:37:37.009388 8476 ts_tablet_manager.cc:1428] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260926 21:37:37.009969 8474 ts_tablet_manager.cc:1428] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
W20260926 21:37:37.047744 8172 tablet.cc:2369] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:37:37.065927 8311 tablet.cc:2369] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:37:37.098415 8450 tablet.cc:2369] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:37:37.137423 8480 raft_consensus.cc:482] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:37.137761 8480 raft_consensus.cc:504] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } }
I20260926 21:37:37.139521 8480 leader_election.cc:288] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca564b7ce4b046999d826c988f873132 (127.0.97.3:41235), 987167ed599c4851974443caa40d7224 (127.0.97.1:36801)
I20260926 21:37:37.146195 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.140728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45345 (local address 127.0.97.1:36801)
0926 21:37:37.140925 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.140932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.141154 (+ 222us) server_negotiation.cc:407] Connection header received
0926 21:37:37.141359 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.141363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.141430 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.141525 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:37.142050 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.142724 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.144451 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.145259 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.145405 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.145511 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.145826 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.145948 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:37:37.146389 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.140030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50849 (local address 127.0.97.3:41235)
0926 21:37:37.140254 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.140261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.140514 (+ 253us) server_negotiation.cc:407] Connection header received
0926 21:37:37.140793 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.140802 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.140884 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.140954 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:37.141689 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.142402 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.144522 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.145413 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.145562 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.145675 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.145979 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.146104 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:37:37.146394 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.140461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36801 (local address 127.0.97.2:45345)
0926 21:37:37.141056 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:37.141069 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:37.141168 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:37.141543 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:37.141547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:37.141571 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:37.141905 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.141913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.142854 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.142860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.144299 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.144310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.145465 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.145471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.145641 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.145683 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 21:37:37.145800 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 21:37:37.145928 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":3,"thread_start_us":314,"threads_started":1}
I20260926 21:37:37.146507 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.139706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41235 (local address 127.0.97.2:50849)
0926 21:37:37.140378 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:37.140398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:37.140590 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:37.141174 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:37.141182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:37.141246 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:37.141544 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.141552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.142539 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.142545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.144285 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.144306 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.145644 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.145651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.145828 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.145850 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:37:37.145953 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 21:37:37.146081 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":87,"thread_start_us":298,"threads_started":1}
I20260926 21:37:37.147603 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eae0bf12ddc4bff939558bbaa6176e0" candidate_uuid: "c985c57f3f374b39adc4e704aaa782a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca564b7ce4b046999d826c988f873132" is_pre_election: true
I20260926 21:37:37.147631 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eae0bf12ddc4bff939558bbaa6176e0" candidate_uuid: "c985c57f3f374b39adc4e704aaa782a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224" is_pre_election: true
I20260926 21:37:37.148136 8399 raft_consensus.cc:2454] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c985c57f3f374b39adc4e704aaa782a8 in term 0.
I20260926 21:37:37.148159 8121 raft_consensus.cc:2454] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c985c57f3f374b39adc4e704aaa782a8 in term 0.
I20260926 21:37:37.148788 8192 leader_election.cc:302] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c985c57f3f374b39adc4e704aaa782a8, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:37.149173 8480 raft_consensus.cc:2781] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:37:37.149398 8480 raft_consensus.cc:482] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:37:37.149531 8480 raft_consensus.cc:3037] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:37.151661 8480 raft_consensus.cc:504] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } }
I20260926 21:37:37.152223 8480 leader_election.cc:288] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [CANDIDATE]: Term 1 election: Requested vote from peers ca564b7ce4b046999d826c988f873132 (127.0.97.3:41235), 987167ed599c4851974443caa40d7224 (127.0.97.1:36801)
I20260926 21:37:37.152729 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eae0bf12ddc4bff939558bbaa6176e0" candidate_uuid: "c985c57f3f374b39adc4e704aaa782a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca564b7ce4b046999d826c988f873132"
I20260926 21:37:37.152828 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eae0bf12ddc4bff939558bbaa6176e0" candidate_uuid: "c985c57f3f374b39adc4e704aaa782a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224"
I20260926 21:37:37.152933 8399 raft_consensus.cc:3037] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:37.153069 8121 raft_consensus.cc:3037] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:37.155115 8399 raft_consensus.cc:2454] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c985c57f3f374b39adc4e704aaa782a8 in term 1.
I20260926 21:37:37.155221 8121 raft_consensus.cc:2454] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c985c57f3f374b39adc4e704aaa782a8 in term 1.
I20260926 21:37:37.155507 8192 leader_election.cc:302] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c985c57f3f374b39adc4e704aaa782a8, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:37.155786 8480 raft_consensus.cc:2781] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:37.156236 8480 raft_consensus.cc:686] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 1 LEADER]: Becoming Leader. State: Replica: c985c57f3f374b39adc4e704aaa782a8, State: Running, Role: LEADER
I20260926 21:37:37.156780 8480 consensus_queue.cc:237] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } }
I20260926 21:37:37.161263 7964 catalog_manager.cc:5495] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c985c57f3f374b39adc4e704aaa782a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } health_report { overall_health: UNKNOWN } } }
I20260926 21:37:37.192430 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.187654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.0.97.2:39219)
0926 21:37:37.187846 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.187850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.187868 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:37:37.187998 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.188002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.188075 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.188196 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:37:37.188497 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.189246 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.190540 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.190931 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.191019 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.191375 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:37:37.191878 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:37:37.191984 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.192060 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.192121 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:37:37.200475 8121 raft_consensus.cc:1261] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Refusing update from remote peer c985c57f3f374b39adc4e704aaa782a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:37.200526 8399 raft_consensus.cc:1261] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Refusing update from remote peer c985c57f3f374b39adc4e704aaa782a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:37.201445 8484 consensus_queue.cc:1035] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:37.201803 8480 consensus_queue.cc:1035] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:37.228224 7963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:37.228534 7963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:37.232293 7963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7: Sending DeleteTablet for 3 replicas of tablet 3eae0bf12ddc4bff939558bbaa6176e0
I20260926 21:37:37.233541 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eae0bf12ddc4bff939558bbaa6176e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58958
I20260926 21:37:37.233551 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eae0bf12ddc4bff939558bbaa6176e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55786
I20260926 21:37:37.233742 8101 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eae0bf12ddc4bff939558bbaa6176e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34874
I20260926 21:37:37.235375 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:37:37.236182 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:37:37.238085 8496 tablet_replica.cc:330] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: stopping tablet replica
I20260926 21:37:37.238552 8496 raft_consensus.cc:2229] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:37.239111 8496 raft_consensus.cc:2258] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:37.239897 8498 tablet_replica.cc:330] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: stopping tablet replica
I20260926 21:37:37.240319 8498 raft_consensus.cc:2229] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:37.242497 8498 raft_consensus.cc:2258] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:37.245079 8101 tablet_service.cc:1467] Processing CreateTablet for tablet ecfbc5aa082e4e65a782aac5bed06505 (DEFAULT_TABLE table=test-table [id=d220cd1489814bda824d5094f9360598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:37.245643 8101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecfbc5aa082e4e65a782aac5bed06505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:37.245834 8239 tablet_service.cc:1467] Processing CreateTablet for tablet ecfbc5aa082e4e65a782aac5bed06505 (DEFAULT_TABLE table=test-table [id=d220cd1489814bda824d5094f9360598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:37.246382 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecfbc5aa082e4e65a782aac5bed06505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:37.250476 8379 tablet_service.cc:1467] Processing CreateTablet for tablet ecfbc5aa082e4e65a782aac5bed06505 (DEFAULT_TABLE table=test-table [id=d220cd1489814bda824d5094f9360598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:37.250945 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecfbc5aa082e4e65a782aac5bed06505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:37.253486 8476 tablet_bootstrap.cc:492] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: Bootstrap starting.
I20260926 21:37:37.255121 8476 tablet_bootstrap.cc:654] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:37.258237 8474 tablet_bootstrap.cc:492] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: Bootstrap starting.
I20260926 21:37:37.259758 8498 ts_tablet_manager.cc:1905] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:37.259945 8474 tablet_bootstrap.cc:654] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:37.260223 8496 ts_tablet_manager.cc:1905] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:37.260564 8497 tablet_replica.cc:330] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: stopping tablet replica
I20260926 21:37:37.261930 8473 tablet_bootstrap.cc:492] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: Bootstrap starting.
I20260926 21:37:37.262472 8497 raft_consensus.cc:2229] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:37:37.263273 8497 raft_consensus.cc:2258] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:37.264101 8473 tablet_bootstrap.cc:654] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:37.267315 8497 ts_tablet_manager.cc:1905] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:37.279151 8496 ts_tablet_manager.cc:1918] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:37.279397 8496 log.cc:1196] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/3eae0bf12ddc4bff939558bbaa6176e0
I20260926 21:37:37.279986 8496 ts_tablet_manager.cc:1939] T 3eae0bf12ddc4bff939558bbaa6176e0 P ca564b7ce4b046999d826c988f873132: Deleting consensus metadata
I20260926 21:37:37.282362 8473 tablet_bootstrap.cc:492] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: No bootstrap required, opened a new log
I20260926 21:37:37.282486 8473 ts_tablet_manager.cc:1397] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.008s
I20260926 21:37:37.283083 8473 raft_consensus.cc:348] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.283239 8473 raft_consensus.cc:374] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:37.283295 8473 raft_consensus.cc:729] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca564b7ce4b046999d826c988f873132, State: Initialized, Role: FOLLOWER
I20260926 21:37:37.283542 8473 consensus_queue.cc:260] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.284092 8473 ts_tablet_manager.cc:1428] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 21:37:37.287901 7949 catalog_manager.cc:4841] TS ca564b7ce4b046999d826c988f873132 (127.0.97.3:41235): tablet 3eae0bf12ddc4bff939558bbaa6176e0 (table test-table [id=52779608ab8a42d283171597d999b98a]) successfully deleted
I20260926 21:37:37.292294 8497 ts_tablet_manager.cc:1918] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:37.292495 8497 log.cc:1196] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/3eae0bf12ddc4bff939558bbaa6176e0
I20260926 21:37:37.293085 8497 ts_tablet_manager.cc:1939] T 3eae0bf12ddc4bff939558bbaa6176e0 P c985c57f3f374b39adc4e704aaa782a8: Deleting consensus metadata
I20260926 21:37:37.293574 8498 ts_tablet_manager.cc:1918] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:37.293776 8498 log.cc:1196] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/3eae0bf12ddc4bff939558bbaa6176e0
I20260926 21:37:37.294203 8498 ts_tablet_manager.cc:1939] T 3eae0bf12ddc4bff939558bbaa6176e0 P 987167ed599c4851974443caa40d7224: Deleting consensus metadata
I20260926 21:37:37.295017 7951 catalog_manager.cc:4841] TS c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219): tablet 3eae0bf12ddc4bff939558bbaa6176e0 (table test-table [id=52779608ab8a42d283171597d999b98a]) successfully deleted
I20260926 21:37:37.295783 7949 catalog_manager.cc:4841] TS 987167ed599c4851974443caa40d7224 (127.0.97.1:36801): tablet 3eae0bf12ddc4bff939558bbaa6176e0 (table test-table [id=52779608ab8a42d283171597d999b98a]) successfully deleted
I20260926 21:37:37.295975 8474 tablet_bootstrap.cc:492] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: No bootstrap required, opened a new log
I20260926 21:37:37.296129 8474 ts_tablet_manager.cc:1397] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.013s
I20260926 21:37:37.296615 8476 tablet_bootstrap.cc:492] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: No bootstrap required, opened a new log
I20260926 21:37:37.296783 8476 ts_tablet_manager.cc:1397] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.015s
I20260926 21:37:37.297008 8474 raft_consensus.cc:348] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.297263 8474 raft_consensus.cc:374] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:37.297369 8474 raft_consensus.cc:729] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c985c57f3f374b39adc4e704aaa782a8, State: Initialized, Role: FOLLOWER
I20260926 21:37:37.297547 8476 raft_consensus.cc:348] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.297621 8474 consensus_queue.cc:260] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.297751 8476 raft_consensus.cc:374] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:37.297843 8476 raft_consensus.cc:729] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987167ed599c4851974443caa40d7224, State: Initialized, Role: FOLLOWER
I20260926 21:37:37.298151 8476 consensus_queue.cc:260] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.298283 8474 ts_tablet_manager.cc:1428] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 21:37:37.298830 8476 ts_tablet_manager.cc:1428] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 21:37:37.429132 8479 raft_consensus.cc:482] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:37.429418 8479 raft_consensus.cc:504] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.431088 8479 leader_election.cc:288] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 987167ed599c4851974443caa40d7224 (127.0.97.1:36801), c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219)
I20260926 21:37:37.439312 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.431482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37645 (local address 127.0.97.2:39219)
0926 21:37:37.431676 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.431686 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.432981 (+ 1295us) server_negotiation.cc:407] Connection header received
0926 21:37:37.433100 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.433106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.433208 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.433324 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:37.434027 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.435081 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.436907 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.438189 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.438350 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.438451 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.438830 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.438977 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:37:37.439505 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.431901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55953 (local address 127.0.97.1:36801)
0926 21:37:37.432121 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.432129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.432757 (+ 628us) server_negotiation.cc:407] Connection header received
0926 21:37:37.433075 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.433082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.433153 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.433310 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:37:37.433712 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.434812 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.437300 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.438570 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.438715 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.438820 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.439111 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.439245 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:37:37.439554 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.431342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36801 (local address 127.0.97.3:55953)
0926 21:37:37.432566 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:37.432589 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:37.432869 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:37.433331 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:37.433335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:37.433358 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:37.433557 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.433570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.434965 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.434970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.437097 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.437117 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.438796 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.438803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.438959 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.438981 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:37:37.439087 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 21:37:37.439227 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"thread_start_us":1216,"threads_started":1}
I20260926 21:37:37.440284 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.431422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39219 (local address 127.0.97.3:37645)
0926 21:37:37.432873 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:37.432888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:37.432987 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:37.433484 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:37.433489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:37.433519 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:37:37.433833 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.433844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.435203 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.435210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.436733 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.436747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.438397 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.438404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.438601 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.438644 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 21:37:37.439790 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0926 21:37:37.439933 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"mutex_wait_us":50,"thread_start_us":885,"threads_started":1}
I20260926 21:37:37.440213 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfbc5aa082e4e65a782aac5bed06505" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224" is_pre_election: true
I20260926 21:37:37.440486 8121 raft_consensus.cc:2454] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 0.
I20260926 21:37:37.441018 8331 leader_election.cc:302] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 987167ed599c4851974443caa40d7224, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:37.441457 8479 raft_consensus.cc:2781] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:37:37.441289 8264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfbc5aa082e4e65a782aac5bed06505" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c985c57f3f374b39adc4e704aaa782a8" is_pre_election: true
I20260926 21:37:37.441588 8479 raft_consensus.cc:482] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:37:37.441681 8479 raft_consensus.cc:3037] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:37.441795 8264 raft_consensus.cc:2454] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 0.
I20260926 21:37:37.443987 8479 raft_consensus.cc:504] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.444630 8479 leader_election.cc:288] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 election: Requested vote from peers 987167ed599c4851974443caa40d7224 (127.0.97.1:36801), c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219)
I20260926 21:37:37.445102 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfbc5aa082e4e65a782aac5bed06505" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224"
I20260926 21:37:37.445135 8264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfbc5aa082e4e65a782aac5bed06505" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c985c57f3f374b39adc4e704aaa782a8"
I20260926 21:37:37.445369 8121 raft_consensus.cc:3037] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:37.445384 8264 raft_consensus.cc:3037] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:37.447358 8121 raft_consensus.cc:2454] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 1.
I20260926 21:37:37.447785 8331 leader_election.cc:302] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 987167ed599c4851974443caa40d7224, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:37.448071 8479 raft_consensus.cc:2781] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:37.448563 8479 raft_consensus.cc:686] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 1 LEADER]: Becoming Leader. State: Replica: ca564b7ce4b046999d826c988f873132, State: Running, Role: LEADER
I20260926 21:37:37.448666 8264 raft_consensus.cc:2454] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 1.
I20260926 21:37:37.448977 8479 consensus_queue.cc:237] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.452766 7963 catalog_manager.cc:5495] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca564b7ce4b046999d826c988f873132 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ca564b7ce4b046999d826c988f873132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } health_report { overall_health: HEALTHY } } }
Sep 26 21:37:37 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458655, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:37.552595 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.523794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35409 (local address 127.0.0.1:45646)
0926 21:37:37.524499 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:37.524524 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:37.524692 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:37.530258 (+ 5566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:37.530262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:37.531127 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:37.531437 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.531449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.533048 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.533052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.534059 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.534070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.535055 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.535059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.535247 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.543704 (+ 8457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:37.543738 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:37.546095 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:37.550147 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:37.550159 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:37.550165 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:37.550588 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:37.551013 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:37.551016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:37.551018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:37.551176 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:37.551816 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:37.551823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:37.551974 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 21:37:37.552106 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 21:37:37.552310 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":38,"thread_start_us":335,"threads_started":1}
I20260926 21:37:37.552963 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.523794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.0.97.62:35409)
0926 21:37:37.529751 (+ 5957us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.529760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.529794 (+ 34us) server_negotiation.cc:407] Connection header received
0926 21:37:37.529867 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.529873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.530027 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.530182 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:37.531627 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.532841 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.534216 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.534893 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.535040 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.546404 (+ 11364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:37.546413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:37.546418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:37.546461 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:37.549983 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:37.550724 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:37.550730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:37.550734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:37.550898 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:37.551289 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:37.551294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:37.551297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:37.551685 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:37.551865 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.552239 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.552469 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":5777,"thread_start_us":5623,"threads_started":1}
I20260926 21:37:37.558077 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45646:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 21:37:37.559151 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:37:37.566435 8101 tablet_service.cc:1467] Processing CreateTablet for tablet 94a275988dd5404a8dab32a230ae697b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c20312bc601b4d3db8ebc4d99f02032d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:37:37.566453 8240 tablet_service.cc:1467] Processing CreateTablet for tablet 94a275988dd5404a8dab32a230ae697b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c20312bc601b4d3db8ebc4d99f02032d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:37:37.567085 8101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a275988dd5404a8dab32a230ae697b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:37.567072 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a275988dd5404a8dab32a230ae697b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:37.572475 8379 tablet_service.cc:1467] Processing CreateTablet for tablet 94a275988dd5404a8dab32a230ae697b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c20312bc601b4d3db8ebc4d99f02032d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 21:37:37.573084 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a275988dd5404a8dab32a230ae697b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:37.576030 8474 tablet_bootstrap.cc:492] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: Bootstrap starting.
I20260926 21:37:37.576221 8476 tablet_bootstrap.cc:492] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: Bootstrap starting.
I20260926 21:37:37.578038 8476 tablet_bootstrap.cc:654] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:37.578080 8474 tablet_bootstrap.cc:654] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:37.581733 8473 tablet_bootstrap.cc:492] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: Bootstrap starting.
I20260926 21:37:37.583709 8473 tablet_bootstrap.cc:654] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:37.592355 8476 tablet_bootstrap.cc:492] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: No bootstrap required, opened a new log
I20260926 21:37:37.592576 8476 ts_tablet_manager.cc:1397] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260926 21:37:37.593354 8476 raft_consensus.cc:348] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.593658 8476 raft_consensus.cc:374] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:37.593751 8476 raft_consensus.cc:729] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987167ed599c4851974443caa40d7224, State: Initialized, Role: FOLLOWER
I20260926 21:37:37.594029 8476 consensus_queue.cc:260] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.594233 8473 tablet_bootstrap.cc:492] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: No bootstrap required, opened a new log
I20260926 21:37:37.594393 8473 ts_tablet_manager.cc:1397] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260926 21:37:37.594481 8481 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.594828 8481 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:37:37.595149 8474 tablet_bootstrap.cc:492] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: No bootstrap required, opened a new log
I20260926 21:37:37.595181 8473 raft_consensus.cc:348] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.595410 8473 raft_consensus.cc:374] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:37.595498 8473 raft_consensus.cc:729] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca564b7ce4b046999d826c988f873132, State: Initialized, Role: FOLLOWER
I20260926 21:37:37.595458 8474 ts_tablet_manager.cc:1397] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.003s
I20260926 21:37:37.595738 8473 consensus_queue.cc:260] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.596321 8473 ts_tablet_manager.cc:1428] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 21:37:37.597141 8479 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.597450 8479 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:37:37.597610 8474 raft_consensus.cc:348] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.597916 8474 raft_consensus.cc:374] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:37.598042 8474 raft_consensus.cc:729] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c985c57f3f374b39adc4e704aaa782a8, State: Initialized, Role: FOLLOWER
I20260926 21:37:37.598395 8476 ts_tablet_manager.cc:1428] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 21:37:37.598364 8474 consensus_queue.cc:260] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.599113 8474 ts_tablet_manager.cc:1428] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 21:37:37.599220 8480 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.599488 8480 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:37:37.713634 8480 raft_consensus.cc:482] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:37.713888 8480 raft_consensus.cc:504] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.714761 8480 leader_election.cc:288] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 987167ed599c4851974443caa40d7224 (127.0.97.1:36801), ca564b7ce4b046999d826c988f873132 (127.0.97.3:41235)
I20260926 21:37:37.715341 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a275988dd5404a8dab32a230ae697b" candidate_uuid: "c985c57f3f374b39adc4e704aaa782a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca564b7ce4b046999d826c988f873132" is_pre_election: true
I20260926 21:37:37.715586 8399 raft_consensus.cc:2454] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c985c57f3f374b39adc4e704aaa782a8 in term 0.
I20260926 21:37:37.716218 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a275988dd5404a8dab32a230ae697b" candidate_uuid: "c985c57f3f374b39adc4e704aaa782a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224" is_pre_election: true
I20260926 21:37:37.716522 8121 raft_consensus.cc:2454] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c985c57f3f374b39adc4e704aaa782a8 in term 0.
I20260926 21:37:37.716522 8192 leader_election.cc:302] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c985c57f3f374b39adc4e704aaa782a8, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:37.717235 8480 raft_consensus.cc:2781] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:37:37.717377 8480 raft_consensus.cc:482] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:37:37.717494 8480 raft_consensus.cc:3037] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:37.720767 8480 raft_consensus.cc:504] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.721509 8480 leader_election.cc:288] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [CANDIDATE]: Term 1 election: Requested vote from peers 987167ed599c4851974443caa40d7224 (127.0.97.1:36801), ca564b7ce4b046999d826c988f873132 (127.0.97.3:41235)
I20260926 21:37:37.722362 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a275988dd5404a8dab32a230ae697b" candidate_uuid: "c985c57f3f374b39adc4e704aaa782a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca564b7ce4b046999d826c988f873132"
I20260926 21:37:37.722370 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a275988dd5404a8dab32a230ae697b" candidate_uuid: "c985c57f3f374b39adc4e704aaa782a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224"
I20260926 21:37:37.722595 8399 raft_consensus.cc:3037] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:37.722620 8121 raft_consensus.cc:3037] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:37.725607 8399 raft_consensus.cc:2454] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c985c57f3f374b39adc4e704aaa782a8 in term 1.
I20260926 21:37:37.725906 8121 raft_consensus.cc:2454] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c985c57f3f374b39adc4e704aaa782a8 in term 1.
I20260926 21:37:37.726102 8192 leader_election.cc:302] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c985c57f3f374b39adc4e704aaa782a8, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:37.726428 8480 raft_consensus.cc:2781] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:37.726881 8480 raft_consensus.cc:686] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [term 1 LEADER]: Becoming Leader. State: Replica: c985c57f3f374b39adc4e704aaa782a8, State: Running, Role: LEADER
I20260926 21:37:37.727465 8480 consensus_queue.cc:237] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:37.729805 8516 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: TxnStatusTablet state changed. Reason: New leader c985c57f3f374b39adc4e704aaa782a8. Latest consensus state: current_term: 1 leader_uuid: "c985c57f3f374b39adc4e704aaa782a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.730115 8516 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:37:37.731477 7963 catalog_manager.cc:5495] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c985c57f3f374b39adc4e704aaa782a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } health_report { overall_health: UNKNOWN } } }
I20260926 21:37:37.733457 8518 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 21:37:37.733700 8518 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 21:37:37.757372 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.752096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39219 (local address 127.0.0.1:58976)
0926 21:37:37.752376 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:37.752393 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:37.752560 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:37.752981 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:37.752986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:37.752998 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:37:37.753250 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.753261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.754317 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.754322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.755317 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.755328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.756073 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.756079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.756269 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.756325 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:37:37.756734 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:37:37.756756 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:37:37.756855 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 21:37:37.756999 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 21:37:37.757702 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.752103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.0.97.2:39219)
0926 21:37:37.752314 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.752320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.752469 (+ 149us) server_negotiation.cc:407] Connection header received
0926 21:37:37.752749 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.752754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.752853 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.752952 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:37:37.753396 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.754172 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.755468 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.755922 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.756060 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.756511 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:37:37.756625 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:37:37.756718 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.756875 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.756950 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 21:37:37.761549 8399 raft_consensus.cc:1261] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Refusing update from remote peer c985c57f3f374b39adc4e704aaa782a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:37.761508 8121 raft_consensus.cc:1261] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Refusing update from remote peer c985c57f3f374b39adc4e704aaa782a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:37.762166 8516 consensus_queue.cc:1035] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:37.762493 8480 consensus_queue.cc:1035] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:37.771782 8481 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: TxnStatusTablet state changed. Reason: New leader c985c57f3f374b39adc4e704aaa782a8. Latest consensus state: current_term: 1 leader_uuid: "c985c57f3f374b39adc4e704aaa782a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.772142 8481 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:37:37.772465 8520 mvcc.cc:205] Tried to move back new op lower bound from 7333718662185033728 to 7333718662054461440. Current Snapshot: MvccSnapshot[applied={T|T < 7333718662185033728}]
I20260926 21:37:37.773463 8516 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c985c57f3f374b39adc4e704aaa782a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.773808 8516 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:37:37.780292 8481 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c985c57f3f374b39adc4e704aaa782a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.780584 8481 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P 987167ed599c4851974443caa40d7224: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:37:37.784799 8479 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: TxnStatusTablet state changed. Reason: New leader c985c57f3f374b39adc4e704aaa782a8. Latest consensus state: current_term: 1 leader_uuid: "c985c57f3f374b39adc4e704aaa782a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.785099 8479 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:37:37.786190 8480 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c985c57f3f374b39adc4e704aaa782a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.786499 8480 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P c985c57f3f374b39adc4e704aaa782a8: This TxnStatusTablet replica's current role is: LEADER
I20260926 21:37:37.795123 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.782201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.0.97.3:41235)
0926 21:37:37.783230 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.783237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.783548 (+ 311us) server_negotiation.cc:407] Connection header received
0926 21:37:37.783775 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.783780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.783919 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.784116 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:37:37.784778 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.787869 (+ 3091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.791192 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.792167 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.792326 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.793080 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:37:37.793905 (+ 825us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:37:37.794046 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.794671 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.794759 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":833,"threads_started":1}
I20260926 21:37:37.801031 8479 tablet_replica.cc:439] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c985c57f3f374b39adc4e704aaa782a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } } }
I20260926 21:37:37.801376 8479 tablet_replica.cc:442] T 94a275988dd5404a8dab32a230ae697b P ca564b7ce4b046999d826c988f873132: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 21:37:37.815248 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.808346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47519 (local address 127.0.97.2:39219)
0926 21:37:37.808513 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.808528 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.808726 (+ 198us) server_negotiation.cc:407] Connection header received
0926 21:37:37.809007 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.809013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.809099 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.809255 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:37:37.809783 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.810924 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.813084 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.813837 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.813963 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.814480 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:37:37.814627 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:37:37.814722 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.814872 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.814957 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:37:37.815536 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.807947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39219 (local address 127.0.97.3:47519)
0926 21:37:37.808594 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:37.808625 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:37.808849 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:37.809275 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:37.809282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:37.809296 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:37:37.809623 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.809634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.811077 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.811083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.812906 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.812922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.814030 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.814037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.814214 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.814292 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:37:37.814741 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:37:37.814753 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 21:37:37.814839 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 21:37:37.814995 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":584,"threads_started":1}
I20260926 21:37:37.826292 8121 raft_consensus.cc:1261] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Refusing update from remote peer ca564b7ce4b046999d826c988f873132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:37.827178 8264 raft_consensus.cc:1261] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Refusing update from remote peer ca564b7ce4b046999d826c988f873132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:37.826990 8504 consensus_queue.cc:1035] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:37.827889 8504 consensus_queue.cc:1035] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:37.832252 8522 mvcc.cc:205] Tried to move back new op lower bound from 7333718662450511872 to 7333718660915466240. Current Snapshot: MvccSnapshot[applied={T|T < 7333718662450511872}]
I20260926 21:37:37.832592 8521 mvcc.cc:205] Tried to move back new op lower bound from 7333718662450511872 to 7333718660915466240. Current Snapshot: MvccSnapshot[applied={T|T < 7333718662450511872}]
I20260926 21:37:37.865790 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.856164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42873 (local address 127.0.97.3:41235)
0926 21:37:37.856338 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:37.856344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:37.858621 (+ 2277us) server_negotiation.cc:407] Connection header received
0926 21:37:37.858818 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:37.858825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:37.858909 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:37.859007 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:37:37.859591 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.860876 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.863277 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.863977 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.864081 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.864876 (+ 795us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:37:37.865008 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:37:37.865066 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:37.865344 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 21:37:37.865428 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 21:37:37.866645 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:37.856488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41235 (local address 127.0.97.2:42873)
0926 21:37:37.858508 (+ 2020us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:37.858527 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:37.858677 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:37.859035 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:37.859040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:37.859053 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 21:37:37.859388 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.859404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.861043 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.861049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.863099 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:37.863113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:37.864308 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:37.864314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:37.864508 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:37.864634 (+ 126us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 21:37:37.865150 (+ 516us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 21:37:37.865161 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 21:37:37.866050 (+ 889us) client_negotiation.cc:240] Negotiation successful
0926 21:37:37.866220 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1807,"thread_start_us":1675,"threads_started":1}
I20260926 21:37:37.913796 7962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:37.914108 7962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:37.918148 7962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7: Sending DeleteTablet for 3 replicas of tablet ecfbc5aa082e4e65a782aac5bed06505
I20260926 21:37:37.919092 8101 tablet_service.cc:1514] Processing DeleteTablet for tablet ecfbc5aa082e4e65a782aac5bed06505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34874
I20260926 21:37:37.919306 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet ecfbc5aa082e4e65a782aac5bed06505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58958
I20260926 21:37:37.919782 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet ecfbc5aa082e4e65a782aac5bed06505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55786
I20260926 21:37:37.953385 8545 tablet_replica.cc:330] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: stopping tablet replica
I20260926 21:37:37.953656 8545 raft_consensus.cc:2229] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:37:37.954274 8545 raft_consensus.cc:2258] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:37.956231 8544 tablet_replica.cc:330] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: stopping tablet replica
I20260926 21:37:37.956485 8544 raft_consensus.cc:2229] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:37.956764 8544 raft_consensus.cc:2258] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:37.958050 8545 ts_tablet_manager.cc:1905] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:37.962342 8544 ts_tablet_manager.cc:1905] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:37.964912 8543 tablet_replica.cc:330] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: stopping tablet replica
I20260926 21:37:37.965127 8543 raft_consensus.cc:2229] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:37.965400 8543 raft_consensus.cc:2258] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:37.966663 8543 ts_tablet_manager.cc:1905] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:37.979245 8544 ts_tablet_manager.cc:1918] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:37:37.979411 8544 log.cc:1196] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/ecfbc5aa082e4e65a782aac5bed06505
I20260926 21:37:37.979957 8544 ts_tablet_manager.cc:1939] T ecfbc5aa082e4e65a782aac5bed06505 P c985c57f3f374b39adc4e704aaa782a8: Deleting consensus metadata
I20260926 21:37:37.981254 7951 catalog_manager.cc:4841] TS c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219): tablet ecfbc5aa082e4e65a782aac5bed06505 (table test-table [id=d220cd1489814bda824d5094f9360598]) successfully deleted
I20260926 21:37:37.986528 8545 ts_tablet_manager.cc:1918] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:37:37.986678 8545 log.cc:1196] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/ecfbc5aa082e4e65a782aac5bed06505
I20260926 21:37:37.987128 8545 ts_tablet_manager.cc:1939] T ecfbc5aa082e4e65a782aac5bed06505 P ca564b7ce4b046999d826c988f873132: Deleting consensus metadata
I20260926 21:37:37.988322 7949 catalog_manager.cc:4841] TS ca564b7ce4b046999d826c988f873132 (127.0.97.3:41235): tablet ecfbc5aa082e4e65a782aac5bed06505 (table test-table [id=d220cd1489814bda824d5094f9360598]) successfully deleted
I20260926 21:37:37.989344 8543 ts_tablet_manager.cc:1918] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:37:37.989488 8543 log.cc:1196] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/ecfbc5aa082e4e65a782aac5bed06505
I20260926 21:37:37.989943 8543 ts_tablet_manager.cc:1939] T ecfbc5aa082e4e65a782aac5bed06505 P 987167ed599c4851974443caa40d7224: Deleting consensus metadata
Sep 26 21:37:37 dist-test-slave-lhhq krb5kdc[7919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260926 21:37:37.990990 7949 catalog_manager.cc:4841] TS 987167ed599c4851974443caa40d7224 (127.0.97.1:36801): tablet ecfbc5aa082e4e65a782aac5bed06505 (table test-table [id=d220cd1489814bda824d5094f9360598]) successfully deleted
Sep 26 21:37:38 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:38 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 21:37:38.014370 8511 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:38.001173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.0.97.62:35409)
0926 21:37:38.001395 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:38.001402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:38.001935 (+ 533us) server_negotiation.cc:407] Connection header received
0926 21:37:38.002195 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:38.002212 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:38.002291 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:38.002365 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:38.004282 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.005468 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.007385 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.008022 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.008147 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:38.014086 (+ 5939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45650: BlockingRecv error: recv got EOF from 127.0.0.1:45650 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 21:37:38 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 21:37:38.036172 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:38.017795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.0.97.62:35409)
0926 21:37:38.017946 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:38.017952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:38.017971 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:37:38.018198 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:38.018211 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:38.018287 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:38.018394 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:38.019921 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.021048 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.022990 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.023656 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.023784 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:38.028956 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:38.028963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:38.028968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:38.029011 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:38.032935 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:38.033850 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:38.033857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:38.033861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:38.034009 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:38.034521 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:38.034525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:38.034529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:38.034892 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:38.035150 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:38.035552 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 21:37:38.035798 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:37:38.040446 7962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:37:38.041282 7962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:37:38.049795 8101 tablet_service.cc:1467] Processing CreateTablet for tablet fd0be1f9cb054b2b83702d345966b83d (DEFAULT_TABLE table=test-table [id=674456b7dd9f4526922aef614135eef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:38.050364 8101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0be1f9cb054b2b83702d345966b83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:38.051363 8240 tablet_service.cc:1467] Processing CreateTablet for tablet fd0be1f9cb054b2b83702d345966b83d (DEFAULT_TABLE table=test-table [id=674456b7dd9f4526922aef614135eef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:38.051800 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0be1f9cb054b2b83702d345966b83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:38.057687 8476 tablet_bootstrap.cc:492] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Bootstrap starting.
I20260926 21:37:38.058024 8379 tablet_service.cc:1467] Processing CreateTablet for tablet fd0be1f9cb054b2b83702d345966b83d (DEFAULT_TABLE table=test-table [id=674456b7dd9f4526922aef614135eef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:38.058439 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0be1f9cb054b2b83702d345966b83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:38.059465 8476 tablet_bootstrap.cc:654] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:38.067049 8474 tablet_bootstrap.cc:492] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: Bootstrap starting.
I20260926 21:37:38.068239 8473 tablet_bootstrap.cc:492] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: Bootstrap starting.
I20260926 21:37:38.068785 8474 tablet_bootstrap.cc:654] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:38.071647 8474 tablet_bootstrap.cc:492] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: No bootstrap required, opened a new log
I20260926 21:37:38.071856 8474 ts_tablet_manager.cc:1397] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 21:37:38.072081 8473 tablet_bootstrap.cc:654] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:38.073882 8474 raft_consensus.cc:348] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.074429 8474 raft_consensus.cc:374] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:38.074586 8474 raft_consensus.cc:729] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c985c57f3f374b39adc4e704aaa782a8, State: Initialized, Role: FOLLOWER
I20260926 21:37:38.074954 8474 consensus_queue.cc:260] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.079056 8474 ts_tablet_manager.cc:1428] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260926 21:37:38.083933 8473 tablet_bootstrap.cc:492] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: No bootstrap required, opened a new log
I20260926 21:37:38.084132 8473 ts_tablet_manager.cc:1397] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.009s
I20260926 21:37:38.084390 8476 tablet_bootstrap.cc:492] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: No bootstrap required, opened a new log
I20260926 21:37:38.084589 8476 ts_tablet_manager.cc:1397] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.009s
I20260926 21:37:38.085091 8473 raft_consensus.cc:348] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.085454 8473 raft_consensus.cc:374] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:38.085569 8473 raft_consensus.cc:729] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca564b7ce4b046999d826c988f873132, State: Initialized, Role: FOLLOWER
I20260926 21:37:38.085511 8476 raft_consensus.cc:348] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.085906 8476 raft_consensus.cc:374] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:38.085880 8473 consensus_queue.cc:260] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.086035 8476 raft_consensus.cc:729] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987167ed599c4851974443caa40d7224, State: Initialized, Role: FOLLOWER
I20260926 21:37:38.086341 8476 consensus_queue.cc:260] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.086592 8473 ts_tablet_manager.cc:1428] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 21:37:38.086964 8476 ts_tablet_manager.cc:1428] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 21:37:38.119166 8479 raft_consensus.cc:482] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:38.119467 8479 raft_consensus.cc:504] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.120208 8479 leader_election.cc:288] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 987167ed599c4851974443caa40d7224 (127.0.97.1:36801), c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219)
I20260926 21:37:38.120702 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0be1f9cb054b2b83702d345966b83d" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224" is_pre_election: true
I20260926 21:37:38.120970 8120 raft_consensus.cc:2454] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 0.
I20260926 21:37:38.120908 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0be1f9cb054b2b83702d345966b83d" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c985c57f3f374b39adc4e704aaa782a8" is_pre_election: true
I20260926 21:37:38.121218 8263 raft_consensus.cc:2454] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 0.
I20260926 21:37:38.121685 8331 leader_election.cc:302] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 987167ed599c4851974443caa40d7224, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:38.122010 8479 raft_consensus.cc:2781] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:37:38.122135 8479 raft_consensus.cc:482] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:37:38.122270 8479 raft_consensus.cc:3037] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:38.124759 8479 raft_consensus.cc:504] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.125923 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0be1f9cb054b2b83702d345966b83d" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224"
I20260926 21:37:38.125931 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0be1f9cb054b2b83702d345966b83d" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c985c57f3f374b39adc4e704aaa782a8"
I20260926 21:37:38.126134 8120 raft_consensus.cc:3037] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:38.126142 8263 raft_consensus.cc:3037] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:38.129114 8120 raft_consensus.cc:2454] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 1.
I20260926 21:37:38.129474 8479 leader_election.cc:288] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 election: Requested vote from peers 987167ed599c4851974443caa40d7224 (127.0.97.1:36801), c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219)
I20260926 21:37:38.129319 8263 raft_consensus.cc:2454] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 1.
I20260926 21:37:38.129985 8333 leader_election.cc:302] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 987167ed599c4851974443caa40d7224, c985c57f3f374b39adc4e704aaa782a8, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:38.130271 8479 raft_consensus.cc:2781] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:38.130575 8479 raft_consensus.cc:686] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 1 LEADER]: Becoming Leader. State: Replica: ca564b7ce4b046999d826c988f873132, State: Running, Role: LEADER
I20260926 21:37:38.130892 8479 consensus_queue.cc:237] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.135040 7962 catalog_manager.cc:5495] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca564b7ce4b046999d826c988f873132 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ca564b7ce4b046999d826c988f873132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } health_report { overall_health: HEALTHY } } }
I20260926 21:37:38.158615 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:38.153554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.0.97.3:41235)
0926 21:37:38.153729 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:38.153736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:38.153839 (+ 103us) server_negotiation.cc:407] Connection header received
0926 21:37:38.154083 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:38.154087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:38.154159 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:38.154263 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:37:38.154600 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.155395 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.156725 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.157160 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.157388 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:38.157797 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:37:38.157910 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:37:38.158006 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:38.158191 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 21:37:38.158282 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:37:38.161443 8120 raft_consensus.cc:1261] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Refusing update from remote peer ca564b7ce4b046999d826c988f873132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:38.161448 8263 raft_consensus.cc:1261] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Refusing update from remote peer ca564b7ce4b046999d826c988f873132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:38.162086 8540 consensus_queue.cc:1035] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:38.162778 8540 consensus_queue.cc:1035] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:38.184727 7962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:38.185010 7962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:38.188092 7962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7: Sending DeleteTablet for 3 replicas of tablet fd0be1f9cb054b2b83702d345966b83d
I20260926 21:37:38.188884 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0be1f9cb054b2b83702d345966b83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58958
I20260926 21:37:38.189169 8544 tablet_replica.cc:330] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: stopping tablet replica
I20260926 21:37:38.189255 8101 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0be1f9cb054b2b83702d345966b83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34874
I20260926 21:37:38.189411 8544 raft_consensus.cc:2229] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:38.189623 8544 raft_consensus.cc:2258] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:38.189911 8543 tablet_replica.cc:330] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: stopping tablet replica
I20260926 21:37:38.190089 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0be1f9cb054b2b83702d345966b83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55786
I20260926 21:37:38.190438 8545 tablet_replica.cc:330] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: stopping tablet replica
I20260926 21:37:38.190622 8545 raft_consensus.cc:2229] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:37:38.190901 8545 raft_consensus.cc:2258] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:38.191718 7962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 21:37:38.191982 8545 ts_tablet_manager.cc:1905] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 21:37:38.192548 7962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:37:38.195462 8544 ts_tablet_manager.cc:1905] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:38.199658 8545 ts_tablet_manager.cc:1918] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:38.202234 8545 log.cc:1196] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/fd0be1f9cb054b2b83702d345966b83d
I20260926 21:37:38.202688 8545 ts_tablet_manager.cc:1939] T fd0be1f9cb054b2b83702d345966b83d P ca564b7ce4b046999d826c988f873132: Deleting consensus metadata
I20260926 21:37:38.205794 8240 tablet_service.cc:1467] Processing CreateTablet for tablet d928205d7e984b7198b3c2916c0e305e (DEFAULT_TABLE table=test-table [id=0282685ced954558aefe5b0b74512fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:38.206770 8379 tablet_service.cc:1467] Processing CreateTablet for tablet d928205d7e984b7198b3c2916c0e305e (DEFAULT_TABLE table=test-table [id=0282685ced954558aefe5b0b74512fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:38.207243 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d928205d7e984b7198b3c2916c0e305e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:38.207597 8544 ts_tablet_manager.cc:1918] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:38.207795 8544 log.cc:1196] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/fd0be1f9cb054b2b83702d345966b83d
I20260926 21:37:38.208284 8544 ts_tablet_manager.cc:1939] T fd0be1f9cb054b2b83702d345966b83d P c985c57f3f374b39adc4e704aaa782a8: Deleting consensus metadata
I20260926 21:37:38.209816 8101 tablet_service.cc:1467] Processing CreateTablet for tablet d928205d7e984b7198b3c2916c0e305e (DEFAULT_TABLE table=test-table [id=0282685ced954558aefe5b0b74512fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:37:38.210285 8101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d928205d7e984b7198b3c2916c0e305e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:38.211230 8543 raft_consensus.cc:2229] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:38.211812 8543 pending_rounds.cc:62] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Trying to abort 2 pending ops.
I20260926 21:37:38.211956 8543 pending_rounds.cc:68] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333718663822585856 op_type: WRITE_OP write_request { tablet_id: "fd0be1f9cb054b2b83702d345966b83d" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\316\370\340\325\006\"8\n\ntest-admin\022*\n 674456b7dd9f4526922aef614135eef9\020\001\030\001 \001(\001" signature: "<redacted>""c\343\341\277^|\233\210\010\002\237\253\313\356\326SQI\256\n0\376\340\257\230\213\301%\205Y@|\341\301\177\r?\357\313\225\275<\367VOnz\305 \017\0271\275\021\001\225\025c\335Z7Q\t." signing_key_seq_num: 0 } } request_id { client_id: "50c640aba0574a188e4d978f554ed9a0" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 21:37:38.212714 8543 pending_rounds.cc:68] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333718663707201536 op_type: NO_OP noop_request { }
I20260926 21:37:38.212898 8543 raft_consensus.cc:2866] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 21:37:38.213032 8543 raft_consensus.cc:2258] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:38.214039 7951 catalog_manager.cc:4841] TS c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219): tablet fd0be1f9cb054b2b83702d345966b83d (table test-table [id=674456b7dd9f4526922aef614135eef9]) successfully deleted
I20260926 21:37:38.214391 8543 ts_tablet_manager.cc:1905] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:38.209412 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d928205d7e984b7198b3c2916c0e305e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:37:38.220659 8543 ts_tablet_manager.cc:1918] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:37:38.220827 8543 log.cc:1196] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/fd0be1f9cb054b2b83702d345966b83d
I20260926 21:37:38.221225 8473 tablet_bootstrap.cc:492] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: Bootstrap starting.
I20260926 21:37:38.221269 8543 ts_tablet_manager.cc:1939] T fd0be1f9cb054b2b83702d345966b83d P 987167ed599c4851974443caa40d7224: Deleting consensus metadata
I20260926 21:37:38.222321 7949 catalog_manager.cc:4841] TS 987167ed599c4851974443caa40d7224 (127.0.97.1:36801): tablet fd0be1f9cb054b2b83702d345966b83d (table test-table [id=674456b7dd9f4526922aef614135eef9]) successfully deleted
I20260926 21:37:38.223183 8473 tablet_bootstrap.cc:654] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:38.226666 8476 tablet_bootstrap.cc:492] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: Bootstrap starting.
I20260926 21:37:38.228118 8476 tablet_bootstrap.cc:654] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:38.229422 7949 catalog_manager.cc:4841] TS ca564b7ce4b046999d826c988f873132 (127.0.97.3:41235): tablet fd0be1f9cb054b2b83702d345966b83d (table test-table [id=674456b7dd9f4526922aef614135eef9]) successfully deleted
I20260926 21:37:38.233656 8474 tablet_bootstrap.cc:492] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: Bootstrap starting.
I20260926 21:37:38.233981 8473 tablet_bootstrap.cc:492] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: No bootstrap required, opened a new log
I20260926 21:37:38.234207 8473 ts_tablet_manager.cc:1397] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260926 21:37:38.235344 8474 tablet_bootstrap.cc:654] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:38.235246 8473 raft_consensus.cc:348] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.235657 8473 raft_consensus.cc:374] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:38.235797 8473 raft_consensus.cc:729] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca564b7ce4b046999d826c988f873132, State: Initialized, Role: FOLLOWER
I20260926 21:37:38.236131 8473 consensus_queue.cc:260] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.240404 8473 ts_tablet_manager.cc:1428] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260926 21:37:38.242636 8101 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0be1f9cb054b2b83702d345966b83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34874
W20260926 21:37:38.246326 7949 catalog_manager.cc:4808] TS 987167ed599c4851974443caa40d7224 (127.0.97.1:36801): delete failed for tablet fd0be1f9cb054b2b83702d345966b83d because the tablet was not found. No further retry: Not found: Tablet not found: fd0be1f9cb054b2b83702d345966b83d
I20260926 21:37:38.247219 8474 tablet_bootstrap.cc:492] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: No bootstrap required, opened a new log
I20260926 21:37:38.247419 8474 ts_tablet_manager.cc:1397] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20260926 21:37:38.248296 8474 raft_consensus.cc:348] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.248550 8474 raft_consensus.cc:374] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:38.248634 8474 raft_consensus.cc:729] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c985c57f3f374b39adc4e704aaa782a8, State: Initialized, Role: FOLLOWER
I20260926 21:37:38.248941 8474 consensus_queue.cc:260] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.249612 8476 tablet_bootstrap.cc:492] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: No bootstrap required, opened a new log
I20260926 21:37:38.249841 8476 ts_tablet_manager.cc:1397] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.009s
I20260926 21:37:38.250705 8476 raft_consensus.cc:348] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.251008 8476 raft_consensus.cc:374] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:38.251133 8476 raft_consensus.cc:729] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987167ed599c4851974443caa40d7224, State: Initialized, Role: FOLLOWER
I20260926 21:37:38.251451 8476 consensus_queue.cc:260] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.252017 8474 ts_tablet_manager.cc:1428] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 21:37:38.252483 8476 ts_tablet_manager.cc:1428] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 21:37:38.390301 8504 raft_consensus.cc:482] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:37:38.390525 8504 raft_consensus.cc:504] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.391285 8504 leader_election.cc:288] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 987167ed599c4851974443caa40d7224 (127.0.97.1:36801), c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219)
I20260926 21:37:38.391829 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d928205d7e984b7198b3c2916c0e305e" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c985c57f3f374b39adc4e704aaa782a8" is_pre_election: true
I20260926 21:37:38.391883 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d928205d7e984b7198b3c2916c0e305e" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224" is_pre_election: true
I20260926 21:37:38.392071 8263 raft_consensus.cc:2454] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 0.
I20260926 21:37:38.392179 8120 raft_consensus.cc:2454] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 0.
I20260926 21:37:38.392594 8333 leader_election.cc:302] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c985c57f3f374b39adc4e704aaa782a8, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:38.392892 8504 raft_consensus.cc:2781] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:37:38.393028 8504 raft_consensus.cc:482] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:37:38.393148 8504 raft_consensus.cc:3037] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:38.395378 8504 raft_consensus.cc:504] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.396029 8504 leader_election.cc:288] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 election: Requested vote from peers 987167ed599c4851974443caa40d7224 (127.0.97.1:36801), c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219)
I20260926 21:37:38.396524 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d928205d7e984b7198b3c2916c0e305e" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987167ed599c4851974443caa40d7224"
I20260926 21:37:38.396593 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d928205d7e984b7198b3c2916c0e305e" candidate_uuid: "ca564b7ce4b046999d826c988f873132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c985c57f3f374b39adc4e704aaa782a8"
I20260926 21:37:38.396755 8120 raft_consensus.cc:3037] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:38.396773 8263 raft_consensus.cc:3037] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:38.399850 8263 raft_consensus.cc:2454] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 1.
I20260926 21:37:38.400274 8120 raft_consensus.cc:2454] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca564b7ce4b046999d826c988f873132 in term 1.
I20260926 21:37:38.400444 8333 leader_election.cc:302] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c985c57f3f374b39adc4e704aaa782a8, ca564b7ce4b046999d826c988f873132; no voters:
I20260926 21:37:38.400756 8504 raft_consensus.cc:2781] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:38.400945 8504 raft_consensus.cc:686] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 1 LEADER]: Becoming Leader. State: Replica: ca564b7ce4b046999d826c988f873132, State: Running, Role: LEADER
I20260926 21:37:38.401290 8504 consensus_queue.cc:237] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } }
I20260926 21:37:38.404378 7962 catalog_manager.cc:5495] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca564b7ce4b046999d826c988f873132 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ca564b7ce4b046999d826c988f873132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca564b7ce4b046999d826c988f873132" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41235 } health_report { overall_health: HEALTHY } } }
I20260926 21:37:38.468286 8263 raft_consensus.cc:1261] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Refusing update from remote peer ca564b7ce4b046999d826c988f873132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:38.468422 8120 raft_consensus.cc:1261] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Refusing update from remote peer ca564b7ce4b046999d826c988f873132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:37:38.468858 8504 consensus_queue.cc:1035] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c985c57f3f374b39adc4e704aaa782a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:38.469125 8540 consensus_queue.cc:1035] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "987167ed599c4851974443caa40d7224" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:37:38.513432 7962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:38.513661 7962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:37:38.516552 7962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 932af892f0f5478ab4ef158e8eaea7b7: Sending DeleteTablet for 3 replicas of tablet d928205d7e984b7198b3c2916c0e305e
I20260926 21:37:38.517393 8101 tablet_service.cc:1514] Processing DeleteTablet for tablet d928205d7e984b7198b3c2916c0e305e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34874
I20260926 21:37:38.517417 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet d928205d7e984b7198b3c2916c0e305e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58958
I20260926 21:37:38.517686 8543 tablet_replica.cc:330] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: stopping tablet replica
I20260926 21:37:38.517724 8544 tablet_replica.cc:330] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: stopping tablet replica
I20260926 21:37:38.517899 8543 raft_consensus.cc:2229] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:38.517920 8544 raft_consensus.cc:2229] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:37:38.518114 8544 raft_consensus.cc:2258] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:38.518114 8543 raft_consensus.cc:2258] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:38.518272 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet d928205d7e984b7198b3c2916c0e305e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:37:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55786
I20260926 21:37:38.519009 8545 tablet_replica.cc:330] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: stopping tablet replica
I20260926 21:37:38.519140 8544 ts_tablet_manager.cc:1905] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:38.519189 8545 raft_consensus.cc:2229] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:37:38.519497 8545 raft_consensus.cc:2258] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:37:38.519929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 8035
I20260926 21:37:38.520556 8545 ts_tablet_manager.cc:1905] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:38.522449 8543 ts_tablet_manager.cc:1905] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:37:38.526625 8544 ts_tablet_manager.cc:1918] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:37:38.526806 8544 log.cc:1196] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/d928205d7e984b7198b3c2916c0e305e
I20260926 21:37:38.527285 8544 ts_tablet_manager.cc:1939] T d928205d7e984b7198b3c2916c0e305e P c985c57f3f374b39adc4e704aaa782a8: Deleting consensus metadata
I20260926 21:37:38.527328 8545 ts_tablet_manager.cc:1918] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:37:38.527462 8545 log.cc:1196] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/d928205d7e984b7198b3c2916c0e305e
I20260926 21:37:38.527864 8545 ts_tablet_manager.cc:1939] T d928205d7e984b7198b3c2916c0e305e P ca564b7ce4b046999d826c988f873132: Deleting consensus metadata
I20260926 21:37:38.528522 7951 catalog_manager.cc:4841] TS c985c57f3f374b39adc4e704aaa782a8 (127.0.97.2:39219): tablet d928205d7e984b7198b3c2916c0e305e (table test-table [id=0282685ced954558aefe5b0b74512fb5]) successfully deleted
I20260926 21:37:38.529275 7949 catalog_manager.cc:4841] TS ca564b7ce4b046999d826c988f873132 (127.0.97.3:41235): tablet d928205d7e984b7198b3c2916c0e305e (table test-table [id=0282685ced954558aefe5b0b74512fb5]) successfully deleted
I20260926 21:37:38.529548 8543 ts_tablet_manager.cc:1918] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 21:37:38.529695 8543 log.cc:1196] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/d928205d7e984b7198b3c2916c0e305e
I20260926 21:37:38.530128 8543 ts_tablet_manager.cc:1939] T d928205d7e984b7198b3c2916c0e305e P 987167ed599c4851974443caa40d7224: Deleting consensus metadata
I20260926 21:37:38.531024 7949 catalog_manager.cc:4841] TS 987167ed599c4851974443caa40d7224 (127.0.97.1:36801): tablet d928205d7e984b7198b3c2916c0e305e (table test-table [id=0282685ced954558aefe5b0b74512fb5]) successfully deleted
Sep 26 21:37:38 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 21:37:38.544257 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:38.521234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.0.97.1:36801)
0926 21:37:38.529565 (+ 8331us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:38.529573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:38.529593 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:37:38.529649 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:38.529654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:38.529763 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:38.529890 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:38.531265 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.532098 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.533499 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.533907 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.534035 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:38.538398 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:38.538417 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:38.538435 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:38.538468 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:38.541246 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:38.541939 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:38.541949 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:38.541952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:38.542087 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:38.542516 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:38.542521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:38.542530 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:38.542951 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:38.543267 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:38.543651 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 21:37:38.543891 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8200,"spinlock_wait_cycles":2432,"thread_start_us":8175,"threads_started":1}
I20260926 21:37:38.544896 8166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:38.643996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 8035
I20260926 21:37:38.662525 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 8175
Sep 26 21:37:38 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 21:37:38.677619 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:38.663289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.0.97.2:39219)
0926 21:37:38.663773 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:38.663778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:38.663794 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:37:38.663847 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:38.663851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:38.663917 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:38.664007 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:38.665833 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.666532 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.667814 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.668186 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.668278 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:38.672383 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:38.672397 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:38.672412 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:38.672446 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:38.675375 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:38.675957 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:38.675964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:38.675966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:38.676091 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:38.676381 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:38.676384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:38.676386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:38.676703 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:38.676926 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:38.677122 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 21:37:38.677343 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
I20260926 21:37:38.678249 8305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:38.821895 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 8175
I20260926 21:37:38.847182 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 8314
Sep 26 21:37:38 dist-test-slave-lhhq krb5kdc[7919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260926 21:37:38.865235 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:38.847928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.0.97.3:41235)
0926 21:37:38.848409 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:38.848413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:38.848429 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:37:38.848478 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:38.848482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:38.848538 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:38.848624 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:38.850301 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.851014 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.852831 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:38.853321 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:38.853430 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:38.858555 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:38.858576 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:38.858605 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:38.858648 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:38.862421 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:38.863065 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:38.863074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:38.863078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:38.863269 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:38.863600 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:38.863605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:38.863608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:38.864043 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:38.864344 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:38.864534 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 21:37:38.864805 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":300,"threads_started":1}
I20260926 21:37:38.865971 8444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:38.987973 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 8314
I20260926 21:37:39.012302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 7932
I20260926 21:37:39.013698 8004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:39.128140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 7932
2026-09-26T21:37:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5705 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:37:39 dist-test-slave-lhhq krb5kdc[8587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:37:39 dist-test-slave-lhhq krb5kdc[8587](info): set up 2 sockets
Sep 26 21:37:39 dist-test-slave-lhhq krb5kdc[8587](info): commencing operation
krb5kdc: starting...
W20260926 21:37:40.234154 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:37:40 dist-test-slave-lhhq krb5kdc[8587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:37:40.466917 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36129
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:40.795749 8600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:40.796080 8600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:40.796170 8600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:40.803983 8600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:40.804178 8600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:40.804235 8600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:40.804296 8600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:40.804354 8600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:40.814942 8600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:40.816334 8600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:40.817716 8600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:40.830078 8605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:40.830178 8608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:40.830065 8606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:40.830945 8600 server_base.cc:1034] running on GCE node
I20260926 21:37:40.831746 8600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:40.833076 8600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:40.834281 8600 hybrid_clock.cc:648] HybridClock initialized: now 1790458660834240 us; error 38 us; skew 500 ppm
I20260926 21:37:40.834631 8600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T21:37:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1776 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:37:41 dist-test-slave-lhhq krb5kdc[8614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:37:41 dist-test-slave-lhhq krb5kdc[8614](info): set up 2 sockets
Sep 26 21:37:41 dist-test-slave-lhhq krb5kdc[8614](info): commencing operation
krb5kdc: starting...
W20260926 21:37:42.006270 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:37:42 dist-test-slave-lhhq krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:37:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:37:42.232719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35755
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:42.561457 8627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:42.561760 8627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:42.561854 8627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:42.569414 8627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:42.569533 8627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:42.569578 8627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:42.569636 8627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:42.569679 8627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:42.580060 8627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:42.581419 8627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:42.582681 8627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:42.594664 8632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:42.594698 8635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:42.594662 8633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:42.595780 8627 server_base.cc:1034] running on GCE node
I20260926 21:37:42.596612 8627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:42.598169 8627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:42.599376 8627 hybrid_clock.cc:648] HybridClock initialized: now 1790458662599310 us; error 55 us; skew 500 ppm
I20260926 21:37:42.599738 8627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:42 dist-test-slave-lhhq krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:42.609653 8627 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:37:42.611248 8627 webserver.cc:460] Webserver started at http://127.0.97.62:38639/ using document root <none> and password file <none>
I20260926 21:37:42.611994 8627 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:42.612154 8627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:42.612483 8627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:42.615602 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "1699dd839684474490d226089831bcbe"
format_stamp: "Formatted at 2026-09-26 21:37:42 on dist-test-slave-lhhq"
server_key: "0443a157cd136944f84b28cc97a753b8"
server_key_iv: "3dd5827abecbef6fa3447cc5a6076fd8"
server_key_version: "encryptionkey@0"
I20260926 21:37:42.616420 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "1699dd839684474490d226089831bcbe"
format_stamp: "Formatted at 2026-09-26 21:37:42 on dist-test-slave-lhhq"
server_key: "0443a157cd136944f84b28cc97a753b8"
server_key_iv: "3dd5827abecbef6fa3447cc5a6076fd8"
server_key_version: "encryptionkey@0"
I20260926 21:37:42.621802 8627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:37:42.625504 8642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:42.626950 8627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:42.627184 8627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "1699dd839684474490d226089831bcbe"
format_stamp: "Formatted at 2026-09-26 21:37:42 on dist-test-slave-lhhq"
server_key: "0443a157cd136944f84b28cc97a753b8"
server_key_iv: "3dd5827abecbef6fa3447cc5a6076fd8"
server_key_version: "encryptionkey@0"
I20260926 21:37:42.627416 8627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:42.639796 8627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:42.643736 8627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:42.644038 8627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:42.662405 8627 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35755
I20260926 21:37:42.662380 8693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35755 every 8 connection(s)
I20260926 21:37:42.663856 8627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:42.664671 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 8627
I20260926 21:37:42.664951 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:37:42.665684 388 external_mini_cluster.cc:1428] Setting key 2e698b7de739436ed26102e6bd8d7992
I20260926 21:37:42.668342 8694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:37:42 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:42.681654 8694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: Bootstrap starting.
I20260926 21:37:42.685060 8694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: Neither blocks nor log segments found. Creating new log.
I20260926 21:37:42.686149 8694 log.cc:824] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:42.688483 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:42.667607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.0.97.62:35755)
0926 21:37:42.668368 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:42.668380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:42.668420 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:37:42.670094 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:42.670115 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:42.670489 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:42.670850 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:42.672685 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:42.674146 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:42.675913 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:42.676573 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:42.676697 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:42.681343 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:42.681352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:42.681368 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:42.681407 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:42.684349 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:42.685077 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:42.685091 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:42.685097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:42.685332 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:42.685704 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:42.685711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:42.685714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:42.686229 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:42.686489 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:42.687036 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 21:37:42.687504 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":354,"threads_started":1}
I20260926 21:37:42.689245 8694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: No bootstrap required, opened a new log
I20260926 21:37:42.696915 8694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260926 21:37:42.697395 8694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:37:42.697537 8694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1699dd839684474490d226089831bcbe, State: Initialized, Role: FOLLOWER
I20260926 21:37:42.698174 8694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260926 21:37:42.698392 8694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:42.698494 8694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:42.698614 8694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:37:42.700686 8694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260926 21:37:42.701123 8694 leader_election.cc:302] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1699dd839684474490d226089831bcbe; no voters:
I20260926 21:37:42.701742 8694 leader_election.cc:288] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:37:42.701908 8699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:37:42.702617 8699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 1 LEADER]: Becoming Leader. State: Replica: 1699dd839684474490d226089831bcbe, State: Running, Role: LEADER
I20260926 21:37:42.703127 8699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260926 21:37:42.703974 8694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:42.706007 8701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1699dd839684474490d226089831bcbe. Latest consensus state: current_term: 1 leader_uuid: "1699dd839684474490d226089831bcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } } }
I20260926 21:37:42.705971 8700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1699dd839684474490d226089831bcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } } }
I20260926 21:37:42.706434 8701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:42.706530 8700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:42.707976 8704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:42.712376 8704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:42.721310 8704 catalog_manager.cc:1347] Generated new cluster ID: b2c785fdb52247ae936ae2c62792410c
I20260926 21:37:42.721454 8704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:42.738338 8704 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:37:42.739693 8704 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:42.751704 8704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: Generated new TSK 0
I20260926 21:37:42.752611 8704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:37:42.811175 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35755
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:43.149086 8720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:43.149400 8720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:43.149559 8720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:43.156791 8720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:43.156920 8720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:43.157119 8720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:43.167158 8720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:43.168496 8720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:43.169853 8720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:43.183032 8725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:43.183058 8726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:43.183197 8728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:43.183982 8720 server_base.cc:1034] running on GCE node
I20260926 21:37:43.184717 8720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:43.185781 8720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:43.187009 8720 hybrid_clock.cc:648] HybridClock initialized: now 1790458663186950 us; error 59 us; skew 500 ppm
I20260926 21:37:43.187357 8720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:43 dist-test-slave-lhhq krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:43.190615 8720 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:37:43.192071 8720 webserver.cc:460] Webserver started at http://127.0.97.1:34823/ using document root <none> and password file <none>
I20260926 21:37:43.192778 8720 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:43.192934 8720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:43.193311 8720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:43.196375 8720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "83ffc51b742747639b48bac84d1822f3"
format_stamp: "Formatted at 2026-09-26 21:37:43 on dist-test-slave-lhhq"
server_key: "16df47ad2b9c2b5b81a7c2283e700d2a"
server_key_iv: "ef1d3892058e0a73c6be951309f14593"
server_key_version: "encryptionkey@0"
I20260926 21:37:43.197302 8720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "83ffc51b742747639b48bac84d1822f3"
format_stamp: "Formatted at 2026-09-26 21:37:43 on dist-test-slave-lhhq"
server_key: "16df47ad2b9c2b5b81a7c2283e700d2a"
server_key_iv: "ef1d3892058e0a73c6be951309f14593"
server_key_version: "encryptionkey@0"
I20260926 21:37:43.202703 8720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:37:43.206465 8735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:43.207958 8720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:43.208176 8720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "83ffc51b742747639b48bac84d1822f3"
format_stamp: "Formatted at 2026-09-26 21:37:43 on dist-test-slave-lhhq"
server_key: "16df47ad2b9c2b5b81a7c2283e700d2a"
server_key_iv: "ef1d3892058e0a73c6be951309f14593"
server_key_version: "encryptionkey@0"
I20260926 21:37:43.208415 8720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:43.227962 8720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:43.232203 8720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:43.232512 8720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:43.233529 8720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:43.234925 8720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:43.235031 8720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:43.235181 8720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:43.235257 8720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:43.268623 8720 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41535
I20260926 21:37:43.268644 8847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41535 every 8 connection(s)
I20260926 21:37:43.270064 8720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:43.276538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 8720
I20260926 21:37:43.276775 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:37:43.277450 388 external_mini_cluster.cc:1428] Setting key 3cf56d8701b60171ab8de802145a2700
Sep 26 21:37:43 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:43.292424 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:43.272764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46947 (local address 127.0.97.62:35755)
0926 21:37:43.273554 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:43.273559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:43.274487 (+ 928us) server_negotiation.cc:407] Connection header received
0926 21:37:43.276376 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:43.276382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:43.276468 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:43.276601 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:43.278343 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:43.279398 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:43.281958 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:43.282345 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:43.282444 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:43.286612 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:43.286616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:43.286620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:43.286652 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:43.289243 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:43.290036 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:43.290042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:43.290045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:43.290168 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:43.290612 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:43.290617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:43.290620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:43.290901 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:43.291054 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:43.291803 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 21:37:43.292034 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":504,"threads_started":1}
I20260926 21:37:43.293526 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:43.273515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:46947)
0926 21:37:43.274296 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:43.274344 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:43.276081 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:43.276850 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:43.276886 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:43.277494 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:43.278109 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:43.278158 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:43.279560 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:43.279565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:43.281817 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:43.281835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:43.282580 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:43.282588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:43.282926 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:43.283640 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:43.283681 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:43.286355 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:43.289411 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:43.289421 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:43.289440 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:43.289895 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:43.290284 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:43.290289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:43.290293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:43.290491 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:43.291043 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:43.291052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:43.291422 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 21:37:43.291704 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 21:37:43.292284 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":290,"threads_started":1}
I20260926 21:37:43.295239 8848 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35755
I20260926 21:37:43.295604 8848 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:43.296566 8848 heartbeater.cc:511] Master 127.0.97.62:35755 requested a full tablet report, sending...
I20260926 21:37:43.299191 8659 ts_manager.cc:194] Registered new tserver with Master: 83ffc51b742747639b48bac84d1822f3 (127.0.97.1:41535)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:37:43.301030 8659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:37:43.330783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35755
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:43.662712 8854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:43.662999 8854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:43.663169 8854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:43.670588 8854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:43.670698 8854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:43.670893 8854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:43.680675 8854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:43.682121 8854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:43.683460 8854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:43.697243 8860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:43.697221 8862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:43.697156 8859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:43.698065 8854 server_base.cc:1034] running on GCE node
I20260926 21:37:43.698709 8854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:43.699627 8854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:43.700842 8854 hybrid_clock.cc:648] HybridClock initialized: now 1790458663700779 us; error 53 us; skew 500 ppm
I20260926 21:37:43.701248 8854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:43 dist-test-slave-lhhq krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:43.704457 8854 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:37:43.706187 8854 webserver.cc:460] Webserver started at http://127.0.97.2:33279/ using document root <none> and password file <none>
I20260926 21:37:43.706875 8854 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:43.706979 8854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:43.707223 8854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:43.710235 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "40af7ed9f32b4399b61c53e00a358769"
format_stamp: "Formatted at 2026-09-26 21:37:43 on dist-test-slave-lhhq"
server_key: "61a97d0493a5b72092c0d22ff53ee42b"
server_key_iv: "8b2be2dac9693224cda4a204cdf010f5"
server_key_version: "encryptionkey@0"
I20260926 21:37:43.710974 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "40af7ed9f32b4399b61c53e00a358769"
format_stamp: "Formatted at 2026-09-26 21:37:43 on dist-test-slave-lhhq"
server_key: "61a97d0493a5b72092c0d22ff53ee42b"
server_key_iv: "8b2be2dac9693224cda4a204cdf010f5"
server_key_version: "encryptionkey@0"
I20260926 21:37:43.716414 8854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:37:43.720208 8869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:43.722385 8854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 21:37:43.722580 8854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "40af7ed9f32b4399b61c53e00a358769"
format_stamp: "Formatted at 2026-09-26 21:37:43 on dist-test-slave-lhhq"
server_key: "61a97d0493a5b72092c0d22ff53ee42b"
server_key_iv: "8b2be2dac9693224cda4a204cdf010f5"
server_key_version: "encryptionkey@0"
I20260926 21:37:43.722806 8854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:43.751302 8854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:43.756036 8854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:43.756356 8854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:43.757493 8854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:43.758913 8854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:43.759023 8854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:43.759177 8854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:43.759253 8854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:43.793025 8854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38897
I20260926 21:37:43.793067 8981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38897 every 8 connection(s)
I20260926 21:37:43.794394 8854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:43.803545 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 8854
I20260926 21:37:43.803764 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:37:43.804186 388 external_mini_cluster.cc:1428] Setting key 4b83572eb98f9d0ab8eaf805df14ce01
Sep 26 21:37:43 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:43.816619 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:43.797026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56899 (local address 127.0.97.62:35755)
0926 21:37:43.797536 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:43.797543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:43.798623 (+ 1080us) server_negotiation.cc:407] Connection header received
0926 21:37:43.800420 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:43.800424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:43.800492 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:43.800604 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:43.802201 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:43.802941 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:43.804713 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:43.805436 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:43.805549 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:43.809380 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:43.809390 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:43.809397 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:43.809445 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:43.813173 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:43.814067 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:43.814077 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:43.814081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:43.814250 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:43.814692 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:43.814699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:43.814703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:43.815089 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:43.815241 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:43.816160 (+ 919us) server_negotiation.cc:299] Negotiation successful
0926 21:37:43.816358 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":280,"threads_started":1}
I20260926 21:37:43.818264 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:43.797687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:56899)
0926 21:37:43.798441 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:43.798481 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:43.800142 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:43.800814 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:43.800823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:43.801460 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:43.802030 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:43.802046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:43.803079 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:43.803082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:43.804553 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:43.804566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:43.805670 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:43.805678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:43.806002 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:43.806868 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:43.806890 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:43.809081 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:43.813382 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:43.813390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:43.813405 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:43.813897 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:43.814380 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:43.814383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:43.814385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:43.814570 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:43.815255 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:43.815266 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:43.815757 (+ 491us) client_negotiation.cc:769] Sending connection context
0926 21:37:43.816082 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 21:37:43.816686 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":276,"threads_started":1}
I20260926 21:37:43.819839 8982 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35755
I20260926 21:37:43.820181 8982 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:43.820922 8982 heartbeater.cc:511] Master 127.0.97.62:35755 requested a full tablet report, sending...
I20260926 21:37:43.822504 8659 ts_manager.cc:194] Registered new tserver with Master: 40af7ed9f32b4399b61c53e00a358769 (127.0.97.2:38897)
I20260926 21:37:43.823340 8659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56899
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:37:43.856963 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35755
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:44.187721 8988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:44.188062 8988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:44.188235 8988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:44.195698 8988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:44.195878 8988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:44.196054 8988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:44.206315 8988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:44.207695 8988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:44.209043 8988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:44.222486 8996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:44.222512 8994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:44.222489 8993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:44.223769 8988 server_base.cc:1034] running on GCE node
I20260926 21:37:44.224354 8988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:44.225350 8988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:44.226541 8988 hybrid_clock.cc:648] HybridClock initialized: now 1790458664226476 us; error 62 us; skew 500 ppm
I20260926 21:37:44.226912 8988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:44 dist-test-slave-lhhq krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:44.230228 8988 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:37:44.231771 8988 webserver.cc:460] Webserver started at http://127.0.97.3:40963/ using document root <none> and password file <none>
I20260926 21:37:44.232571 8988 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:44.232717 8988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:44.233031 8988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:37:44.236025 8988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "8ed93aad012e4ce3a554f26312b85e70"
format_stamp: "Formatted at 2026-09-26 21:37:44 on dist-test-slave-lhhq"
server_key: "e5036d688ce55d78e16c471992b13bba"
server_key_iv: "9fda1b1fba1c79bc7e3ab0b7ae0bc5e6"
server_key_version: "encryptionkey@0"
I20260926 21:37:44.236825 8988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8ed93aad012e4ce3a554f26312b85e70"
format_stamp: "Formatted at 2026-09-26 21:37:44 on dist-test-slave-lhhq"
server_key: "e5036d688ce55d78e16c471992b13bba"
server_key_iv: "9fda1b1fba1c79bc7e3ab0b7ae0bc5e6"
server_key_version: "encryptionkey@0"
I20260926 21:37:44.242295 8988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:37:44.245942 9003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:44.247318 8988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:44.247512 8988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "8ed93aad012e4ce3a554f26312b85e70"
format_stamp: "Formatted at 2026-09-26 21:37:44 on dist-test-slave-lhhq"
server_key: "e5036d688ce55d78e16c471992b13bba"
server_key_iv: "9fda1b1fba1c79bc7e3ab0b7ae0bc5e6"
server_key_version: "encryptionkey@0"
I20260926 21:37:44.247742 8988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:44.272552 8988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:44.277247 8988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:44.277508 8988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:44.278424 8988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:44.279686 8988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:44.279760 8988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:44.279843 8988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:44.279910 8988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:44.304255 8848 heartbeater.cc:503] Master 127.0.97.62:35755 was elected leader, sending a full tablet report...
I20260926 21:37:44.312557 8988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46669
I20260926 21:37:44.312605 9115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46669 every 8 connection(s)
I20260926 21:37:44.313908 8988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:44.318948 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 8988
I20260926 21:37:44.319146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:37:44.319561 388 external_mini_cluster.cc:1428] Setting key cf294742a6cf7752cb466d33b89b1190
Sep 26 21:37:44 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:37:44.336011 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:44.316783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42693 (local address 127.0.97.62:35755)
0926 21:37:44.317021 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:44.317027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:44.318545 (+ 1518us) server_negotiation.cc:407] Connection header received
0926 21:37:44.322219 (+ 3674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:44.322224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:44.322317 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:44.322424 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:44.323852 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.324533 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.326634 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.327042 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.327137 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:44.330461 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:44.330465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:44.330469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:44.330499 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:44.332864 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:44.333808 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:44.333812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:44.333814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:44.333969 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:44.334470 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:44.334474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:44.334477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:44.334848 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:44.334966 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:44.335607 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 21:37:44.335777 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 21:37:44.337011 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:44.317411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:42693)
0926 21:37:44.318344 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:44.318413 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:44.321933 (+ 3520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:44.322681 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:44.322690 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:44.323175 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:44.323711 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:44.323726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.324653 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.324656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:44.326512 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:44.326524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.327224 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.327228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:44.327431 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:44.328015 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:44.328036 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:44.330261 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:37:44.333026 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:44.333032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:44.333045 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:44.333686 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:44.334097 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:37:44.334102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:37:44.334105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:37:44.334350 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:37:44.334972 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:37:44.334977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:37:44.335288 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 21:37:44.335513 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 21:37:44.335908 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":352,"threads_started":1}
I20260926 21:37:44.338390 9116 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35755
I20260926 21:37:44.338690 9116 heartbeater.cc:464] Registering TS with master...
I20260926 21:37:44.339385 9116 heartbeater.cc:511] Master 127.0.97.62:35755 requested a full tablet report, sending...
I20260926 21:37:44.340747 8659 ts_manager.cc:194] Registered new tserver with Master: 8ed93aad012e4ce3a554f26312b85e70 (127.0.97.3:46669)
I20260926 21:37:44.341576 8659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42693
I20260926 21:37:44.347338 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 21:37:44.487568 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 8720
Sep 26 21:37:44 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:37:44.504252 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:44.488673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.0.97.1:41535)
0926 21:37:44.489322 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:44.489330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:44.489362 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:37:44.489545 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:44.489553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:44.489781 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:44.489941 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:44.491312 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.492340 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.493716 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.494306 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.494455 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:44.498617 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:44.498631 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:44.498645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:44.498680 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:44.501389 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:44.502114 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:44.502121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:44.502124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:44.502241 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:44.502629 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:44.502634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:44.502642 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:44.502969 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:44.503267 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:44.503558 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 21:37:44.503828 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":316,"threads_started":1}
I20260926 21:37:44.505496 8843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:44.587198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 8720
I20260926 21:37:44.603580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 8854
Sep 26 21:37:44 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:37:44.619778 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:44.604492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.0.97.2:38897)
0926 21:37:44.605019 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:44.605026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:44.605049 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:37:44.605105 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:44.605112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:44.605389 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:44.605577 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:44.606791 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.607821 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.609108 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.609788 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.609939 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:44.614131 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:44.614142 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:44.614157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:44.614192 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:44.617073 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:44.617986 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:44.617993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:44.617996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:44.618126 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:44.618472 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:44.618475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:44.618477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:44.618808 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:44.619032 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:44.619273 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 21:37:44.619489 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
I20260926 21:37:44.621047 8977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:44.702970 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 8854
I20260926 21:37:44.719486 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 8988
Sep 26 21:37:44 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:37:44.736176 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:44.720578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.0.97.3:46669)
0926 21:37:44.721338 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:44.721345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:44.721374 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:37:44.721433 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:44.721440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:44.721649 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:44.721780 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:44.723004 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.724078 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.725485 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:44.726040 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:44.726143 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:44.730595 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:44.730610 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:44.730627 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:44.730668 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:44.733732 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:44.734441 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:44.734448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:44.734451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:44.734572 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:44.734906 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:44.734909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:44.734911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:44.735238 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:44.735467 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:44.735698 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 21:37:44.735899 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":345,"threads_started":1}
I20260926 21:37:44.737668 9111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:44.820500 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 8988
I20260926 21:37:44.836570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 8627
I20260926 21:37:44.837770 8689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:37:44.939325 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 8627
I20260926 21:37:44.959717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35755
--webserver_interface=127.0.97.62
--webserver_port=38639
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:37:45.288497 9133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:45.288779 9133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:45.288867 9133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:45.296175 9133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:37:45.296304 9133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:45.296396 9133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:45.296482 9133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:37:45.296566 9133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:37:45.306559 9133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:45.307906 9133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:45.309252 9133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:45.322301 9138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:45.322393 9139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:45.323024 9141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:45.323511 9133 server_base.cc:1034] running on GCE node
I20260926 21:37:45.324337 9133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:45.325861 9133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:45.327059 9133 hybrid_clock.cc:648] HybridClock initialized: now 1790458665327023 us; error 33 us; skew 500 ppm
I20260926 21:37:45.327415 9133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:45 dist-test-slave-lhhq krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:45.330619 9133 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:37:45.332144 9133 webserver.cc:460] Webserver started at http://127.0.97.62:38639/ using document root <none> and password file <none>
I20260926 21:37:45.332870 9133 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:45.332980 9133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:45.338388 9133 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:37:45.341090 9148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:45.342641 9133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:37:45.342855 9133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "1699dd839684474490d226089831bcbe"
format_stamp: "Formatted at 2026-09-26 21:37:42 on dist-test-slave-lhhq"
server_key: "0443a157cd136944f84b28cc97a753b8"
server_key_iv: "3dd5827abecbef6fa3447cc5a6076fd8"
server_key_version: "encryptionkey@0"
I20260926 21:37:45.343851 9133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:45.359411 9133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:45.364050 9133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:45.364359 9133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:45.383680 9133 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35755
I20260926 21:37:45.383699 9199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35755 every 8 connection(s)
I20260926 21:37:45.385356 9133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:37:45.392122 9200 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:37:45.392217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 9133
I20260926 21:37:45.392982 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41535
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35755
--webserver_port=34823
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:37:45.395558 9200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: Bootstrap starting.
I20260926 21:37:45.415093 9200 log.cc:824] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: Log is configured to *not* fsync() on all Append() calls
I20260926 21:37:45.423976 9200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:37:45.424523 9200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: Bootstrap complete.
I20260926 21:37:45.433507 9200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260926 21:37:45.434413 9200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1699dd839684474490d226089831bcbe, State: Initialized, Role: FOLLOWER
I20260926 21:37:45.435050 9200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260926 21:37:45.435309 9200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:37:45.435446 9200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:37:45.435633 9200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:37:45.439538 9200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260926 21:37:45.440078 9200 leader_election.cc:302] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1699dd839684474490d226089831bcbe; no voters:
I20260926 21:37:45.440737 9200 leader_election.cc:288] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 21:37:45.440881 9204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:37:45.442257 9204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [term 2 LEADER]: Becoming Leader. State: Replica: 1699dd839684474490d226089831bcbe, State: Running, Role: LEADER
I20260926 21:37:45.442811 9204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260926 21:37:45.443094 9200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:37:45.445466 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1699dd839684474490d226089831bcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } } }
I20260926 21:37:45.445892 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:45.450961 9206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1699dd839684474490d226089831bcbe. Latest consensus state: current_term: 2 leader_uuid: "1699dd839684474490d226089831bcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699dd839684474490d226089831bcbe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } } }
I20260926 21:37:45.451346 9206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe [sys.catalog]: This master's current role is: LEADER
I20260926 21:37:45.456959 9210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:37:45.463552 9210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:37:45.464931 9210 catalog_manager.cc:1259] Loaded cluster ID: b2c785fdb52247ae936ae2c62792410c
I20260926 21:37:45.465036 9210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:37:45.469023 9210 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:37:45.470460 9210 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1699dd839684474490d226089831bcbe: Loaded TSK: 0
I20260926 21:37:45.472146 9210 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 21:37:45.735323 9202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:45.735599 9202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:45.735743 9202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:45.743814 9202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:45.743925 9202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:45.744089 9202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:37:45.754463 9202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34823
--tserver_master_addrs=127.0.97.62:35755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:45.755775 9202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:45.757079 9202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:45.770219 9227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:45.770184 9226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:45.770717 9229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:45.771067 9202 server_base.cc:1034] running on GCE node
I20260926 21:37:45.771749 9202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:45.772785 9202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:45.774008 9202 hybrid_clock.cc:648] HybridClock initialized: now 1790458665773955 us; error 47 us; skew 500 ppm
I20260926 21:37:45.774349 9202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:45 dist-test-slave-lhhq krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458665, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:45.777554 9202 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 21:37:45.779337 9202 webserver.cc:460] Webserver started at http://127.0.97.1:34823/ using document root <none> and password file <none>
I20260926 21:37:45.780052 9202 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:45.780197 9202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:45.785818 9202 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 21:37:45.788709 9236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:45.790241 9202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:45.790406 9202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "83ffc51b742747639b48bac84d1822f3"
format_stamp: "Formatted at 2026-09-26 21:37:43 on dist-test-slave-lhhq"
server_key: "16df47ad2b9c2b5b81a7c2283e700d2a"
server_key_iv: "ef1d3892058e0a73c6be951309f14593"
server_key_version: "encryptionkey@0"
I20260926 21:37:45.791357 9202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:45.813094 9202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:45.817178 9202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:45.817508 9202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:45.818495 9202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:45.819823 9202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:45.819969 9202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:45.820098 9202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:45.820199 9202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:45.853916 9202 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41535
I20260926 21:37:45.854007 9348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41535 every 8 connection(s)
I20260926 21:37:45.855367 9202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:37:45.865453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 9202
I20260926 21:37:45.866035 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38897
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35755
--webserver_port=33279
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:37:45 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:45 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:45.873533 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:45.858500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:42329)
0926 21:37:45.859246 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:45.859286 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:45.860907 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:45.862234 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:45.862241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:45.862731 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:45.863248 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:45.863265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.864797 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.864801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:45.866193 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:45.866206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.867825 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.867830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:45.868016 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:45.868694 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:45.868720 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:45.872478 (+ 3758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":344,"threads_started":1}
W20260926 21:37:45.874243 9349 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35755 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35755: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
W20260926 21:37:45.874295 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:45.858370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42329 (local address 127.0.97.62:35755)
0926 21:37:45.859117 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:45.859132 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:45.859402 (+ 270us) server_negotiation.cc:407] Connection header received
0926 21:37:45.861439 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:45.861453 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:45.861785 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:45.862089 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:45.863424 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.864656 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.866389 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.867588 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.867784 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:45.873000 (+ 5216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42329: BlockingRecv error: recv got EOF from 127.0.97.1:42329 (error 108)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":231,"threads_started":1}
Sep 26 21:37:45 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:45 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:45.883069 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:45.874810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:58315)
0926 21:37:45.875108 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:45.875124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:45.875209 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:45.875522 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:45.875524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:45.875818 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:45.875953 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:45.875960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.877504 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.877508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:45.878386 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:45.878396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.879394 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.879398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:45.879510 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:45.880133 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:45.880150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:45.882859 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":51}
W20260926 21:37:45.883455 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:45.875005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58315 (local address 127.0.97.62:35755)
0926 21:37:45.875181 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:45.875189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:45.875209 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:37:45.875301 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:45.875309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:45.875387 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:45.875494 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:45.876102 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.877360 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.878527 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.879250 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.879384 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:45.883163 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58315: BlockingRecv error: recv got EOF from 127.0.97.1:58315 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 21:37:45 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:45 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:45.891903 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:45.883809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:54537)
0926 21:37:45.883941 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:45.883961 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:45.884047 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:45.884425 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:45.884428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:45.884687 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:45.884819 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:45.884826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.886379 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.886382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:45.887263 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:45.887273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.888330 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.888334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:45.888461 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:45.889045 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:45.889061 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:45.891693 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 21:37:45.892200 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:45.883909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54537 (local address 127.0.97.62:35755)
0926 21:37:45.884092 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:45.884100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:45.884120 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:37:45.884180 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:45.884186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:45.884283 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:45.884402 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:45.884960 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.886224 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.887402 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:45.888178 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:45.888328 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:45.891947 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54537: BlockingRecv error: recv got EOF from 127.0.97.1:54537 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 21:37:45.892336 9349 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:37:46.197966 9353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:46.198269 9353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:46.198457 9353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:46.205734 9353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:46.205883 9353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:46.206056 9353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:37:46.215797 9353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33279
--tserver_master_addrs=127.0.97.62:35755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:46.217243 9353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:46.218623 9353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:46.232087 9359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:46.232180 9358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:46.232158 9361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:46.233511 9353 server_base.cc:1034] running on GCE node
I20260926 21:37:46.234092 9353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:46.235059 9353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:46.236260 9353 hybrid_clock.cc:648] HybridClock initialized: now 1790458666236213 us; error 37 us; skew 500 ppm
I20260926 21:37:46.236557 9353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458666, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:46.239709 9353 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 21:37:46.241070 9353 webserver.cc:460] Webserver started at http://127.0.97.2:33279/ using document root <none> and password file <none>
I20260926 21:37:46.241806 9353 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:46.241921 9353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:46.247503 9353 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:37:46.250339 9368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:46.251662 9353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:37:46.251834 9353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "40af7ed9f32b4399b61c53e00a358769"
format_stamp: "Formatted at 2026-09-26 21:37:43 on dist-test-slave-lhhq"
server_key: "61a97d0493a5b72092c0d22ff53ee42b"
server_key_iv: "8b2be2dac9693224cda4a204cdf010f5"
server_key_version: "encryptionkey@0"
I20260926 21:37:46.252863 9353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:46.271538 9353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:46.275975 9353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:46.276202 9353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:46.277240 9353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:46.278533 9353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:46.278620 9353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:46.278692 9353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:46.278753 9353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:46.311092 9353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38897
I20260926 21:37:46.311167 9480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38897 every 8 connection(s)
I20260926 21:37:46.312505 9353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:37:46.320595 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 9353
I20260926 21:37:46.321313 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46669
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35755
--webserver_port=40963
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 21:37:46.330782 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:46.314962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34645 (local address 127.0.97.62:35755)
0926 21:37:46.315166 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:46.315173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:46.316271 (+ 1098us) server_negotiation.cc:407] Connection header received
0926 21:37:46.318069 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:46.318073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:46.318137 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:46.318223 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:46.319719 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.320712 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.322485 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.323229 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.323376 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.330587 (+ 7211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34645: BlockingRecv error: recv got EOF from 127.0.97.2:34645 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:37:46.331763 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:46.315452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:34645)
0926 21:37:46.316107 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:46.316149 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:46.317817 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:46.318456 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:46.318464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:46.318949 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:46.319516 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.319534 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.320894 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.320897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.322292 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.322304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.323538 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.323547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.323784 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.325394 (+ 1610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:46.325428 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:46.330379 (+ 4951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":258,"threads_started":1}
W20260926 21:37:46.332589 9481 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35755 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35755: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:46.342101 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:46.333243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:54629)
0926 21:37:46.333516 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:46.333532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:46.333622 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:46.333967 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:46.333970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:46.334274 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:46.334415 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.334423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.335919 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.335923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.336803 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.336813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.337943 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.337948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.338083 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.338767 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:46.338785 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:46.341885 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147}
W20260926 21:37:46.342540 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:46.333406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54629 (local address 127.0.97.62:35755)
0926 21:37:46.333618 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:46.333626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:46.333649 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:37:46.333716 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:46.333724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:46.333810 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:46.333939 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:46.334560 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.335753 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.337012 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.337766 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.337918 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.342244 (+ 4326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54629: BlockingRecv error: recv got EOF from 127.0.97.2:54629 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:46.351579 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:46.342886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:51055)
0926 21:37:46.343178 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:46.343194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:46.343313 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:46.343690 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:46.343693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:46.343961 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:46.344104 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.344112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.345744 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.345748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.346650 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.346660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.347741 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.347745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.347902 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.348531 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:46.348547 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:46.351363 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179}
W20260926 21:37:46.351817 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:46.343051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51055 (local address 127.0.97.62:35755)
0926 21:37:46.343263 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:46.343272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:46.343295 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:37:46.343446 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:46.343453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:46.343545 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:46.343655 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:46.344250 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.345549 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.346817 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.347601 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.347738 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.351594 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51055: BlockingRecv error: recv got EOF from 127.0.97.2:51055 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 21:37:46.351959 9481 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:37:46.659422 9484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:37:46.659701 9484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:37:46.659870 9484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:37:46.667090 9484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:37:46.667207 9484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:37:46.667433 9484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:37:46.677176 9484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40963
--tserver_master_addrs=127.0.97.62:35755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:37:46.678577 9484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:37:46.679893 9484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:37:46.693261 9492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:46.693315 9490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:37:46.693336 9489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:37:46.698635 9484 server_base.cc:1034] running on GCE node
I20260926 21:37:46.699254 9484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:37:46.700213 9484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:37:46.701438 9484 hybrid_clock.cc:648] HybridClock initialized: now 1790458666701407 us; error 54 us; skew 500 ppm
I20260926 21:37:46.701756 9484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458666, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:37:46.704932 9484 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 21:37:46.706519 9484 webserver.cc:460] Webserver started at http://127.0.97.3:40963/ using document root <none> and password file <none>
I20260926 21:37:46.707190 9484 fs_manager.cc:359] Metadata directory not provided
I20260926 21:37:46.707302 9484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:37:46.712654 9484 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:37:46.715423 9499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:46.716784 9484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:37:46.716962 9484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "8ed93aad012e4ce3a554f26312b85e70"
format_stamp: "Formatted at 2026-09-26 21:37:44 on dist-test-slave-lhhq"
server_key: "e5036d688ce55d78e16c471992b13bba"
server_key_iv: "9fda1b1fba1c79bc7e3ab0b7ae0bc5e6"
server_key_version: "encryptionkey@0"
I20260926 21:37:46.718369 9484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:37:46.741572 9484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:37:46.746325 9484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:37:46.746649 9484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:37:46.747641 9484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:37:46.749176 9484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:37:46.749328 9484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 21:37:46.749470 9484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:37:46.749570 9484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:37:46.783094 9484 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46669
I20260926 21:37:46.783187 9611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46669 every 8 connection(s)
I20260926 21:37:46.784637 9484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:37:46.788213 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 9484
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 21:37:46.802322 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:46.787325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55139 (local address 127.0.97.62:35755)
0926 21:37:46.787496 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:46.787501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:46.788999 (+ 1498us) server_negotiation.cc:407] Connection header received
0926 21:37:46.791169 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:46.791174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:46.791260 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:46.791386 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:46.793177 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.793953 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.795676 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.796366 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.796538 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.802013 (+ 5475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55139: BlockingRecv error: recv got EOF from 127.0.97.3:55139 (error 108)
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":59}
I20260926 21:37:46.802910 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:46.787960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:55139)
0926 21:37:46.788798 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:46.788866 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:46.790883 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:46.791684 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:46.791697 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:46.792429 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:46.793005 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.793022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.794113 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.794118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.795523 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.795535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.796626 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.796633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.796887 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.797606 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:46.797635 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:46.801753 (+ 4118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":382,"threads_started":1}
W20260926 21:37:46.803718 9612 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35755 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35755: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260926 21:37:46.806631 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:46.789356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.0.97.62:35755)
0926 21:37:46.791839 (+ 2483us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:46.791860 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:46.791879 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:37:46.791927 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:46.791931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:46.791990 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:46.792105 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:46.793631 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.794404 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.795772 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.796316 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.796458 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.799222 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:37:46.799230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:37:46.799249 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:37:46.799291 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:37:46.802744 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:46.803683 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:46.803693 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:46.803701 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:46.803946 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:37:46.804555 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:37:46.804560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:37:46.804568 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:37:46.805038 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:37:46.805414 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0926 21:37:46.805931 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 21:37:46.806323 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":2375,"thread_start_us":2261,"threads_started":1}
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:46.812434 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:46.804304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:52787)
0926 21:37:46.804551 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:46.804572 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:46.804677 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:46.805010 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:46.805014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:46.805438 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:46.805576 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.805584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.806751 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.806756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.807678 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.807690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.808432 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.808438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.808607 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.809269 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:46.809287 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:46.812222 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260926 21:37:46.812609 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:46.804437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52787 (local address 127.0.97.62:35755)
0926 21:37:46.804610 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:46.804616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:46.804634 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:37:46.804802 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:46.804807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:46.804878 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:46.804985 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:46.805711 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.806615 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.807829 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.808269 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.808408 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.812407 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52787: BlockingRecv error: recv got EOF from 127.0.97.3:52787 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:46.820369 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:46.813158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:59495)
0926 21:37:46.813456 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:46.813475 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:46.813572 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:46.813836 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:46.813839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:46.814193 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:46.814354 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.814362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.815306 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.815309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.816202 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.816212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.816831 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.816835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.816969 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.817541 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:46.817558 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:46.820144 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260926 21:37:46.820637 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:46.813374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59495 (local address 127.0.97.62:35755)
0926 21:37:46.813577 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:46.813581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:46.813597 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:37:46.813663 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:46.813666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:46.813723 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:46.813800 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:46.814477 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.815190 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.816317 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.816705 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.816795 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.820343 (+ 3548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59495: BlockingRecv error: recv got EOF from 127.0.97.3:59495 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260926 21:37:46.820775 9612 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:46 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:46.901742 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:46.893029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:48141)
0926 21:37:46.893577 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:46.893594 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:46.893704 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:46.894073 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:46.894078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:46.894654 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:46.894840 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.894848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.896091 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.896096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.897066 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:46.897077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.897839 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.897845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:46.897982 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.898652 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:46.898670 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:46.901523 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":292,"threads_started":1}
W20260926 21:37:46.901929 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:46.893169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48141 (local address 127.0.97.62:35755)
0926 21:37:46.893357 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:46.893361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:46.893679 (+ 318us) server_negotiation.cc:407] Connection header received
0926 21:37:46.893828 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:46.893833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:46.893913 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:46.894018 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:46.895026 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.895923 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.897230 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:46.897641 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:46.897737 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:46.901723 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48141: BlockingRecv error: recv got EOF from 127.0.97.1:48141 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 21:37:47 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:47 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:47.361742 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:47.352667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:44341)
0926 21:37:47.353264 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:47.353283 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:47.353412 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:47.353747 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:47.353752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:47.354197 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:47.354394 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:47.354404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.355485 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.355489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:47.356528 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:47.356541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.357533 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.357539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:47.357743 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:47.358495 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:47.358513 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:47.361518 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":310,"threads_started":1}
W20260926 21:37:47.362033 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:47.352858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44341 (local address 127.0.97.62:35755)
0926 21:37:47.353106 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:47.353112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:47.353372 (+ 260us) server_negotiation.cc:407] Connection header received
0926 21:37:47.353523 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:47.353527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:47.353591 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:47.353680 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:47.354524 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.355347 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.356731 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.357333 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.357502 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:47.361733 (+ 4231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44341: BlockingRecv error: recv got EOF from 127.0.97.2:44341 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 26 21:37:47 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:47 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:47.830466 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:47.821536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:53159)
0926 21:37:47.822172 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:47.822189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:47.822330 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:47.822645 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:47.822649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:47.823194 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:47.823423 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:47.823432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.824470 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.824475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:47.825726 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:47.825737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.826522 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.826527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:47.826689 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:47.827310 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:47.827327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:47.830207 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":336,"threads_started":1}
W20260926 21:37:47.830612 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:47.821767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53159 (local address 127.0.97.62:35755)
0926 21:37:47.821983 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:47.821990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:47.822290 (+ 300us) server_negotiation.cc:407] Connection header received
0926 21:37:47.822436 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:47.822440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:47.822501 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:47.822585 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:47.823567 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.824316 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.825869 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.826341 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.826477 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:47.830402 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53159: BlockingRecv error: recv got EOF from 127.0.97.3:53159 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 21:37:47 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:47 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:47.913530 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:47.902846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:51713)
0926 21:37:47.903391 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:47.903407 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:47.903574 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:47.904019 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:47.904028 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:47.904630 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:47.904840 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:47.904855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.906334 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.906343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:47.908009 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:47.908028 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.908722 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.908730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:47.908932 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:47.909753 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:47.909782 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:47.913174 (+ 3392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":311,"threads_started":1}
W20260926 21:37:47.913618 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:47.902939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51713 (local address 127.0.97.62:35755)
0926 21:37:47.903158 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:47.903166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:47.903501 (+ 335us) server_negotiation.cc:407] Connection header received
0926 21:37:47.903744 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:47.903749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:47.903842 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:47.903946 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:47.905069 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.906187 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.908174 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:47.908561 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:47.908663 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:47.913395 (+ 4732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51713: BlockingRecv error: recv got EOF from 127.0.97.1:51713 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 21:37:48 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:48 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:48.371414 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:48.362689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:54073)
0926 21:37:48.363268 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:48.363284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:48.363448 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:48.363810 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:48.363815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:48.364239 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:48.364383 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:48.364391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.365484 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.365490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:48.366429 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:48.366440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.367164 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.367170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:48.367350 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:48.367936 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:48.367953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:48.371099 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":311,"threads_started":1}
W20260926 21:37:48.371599 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:48.362780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54073 (local address 127.0.97.62:35755)
0926 21:37:48.363060 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:48.363067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:48.363381 (+ 314us) server_negotiation.cc:407] Connection header received
0926 21:37:48.363589 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:48.363594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:48.363663 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:48.363791 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:48.364538 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.365350 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.366576 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.367001 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.367135 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:48.371313 (+ 4178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54073: BlockingRecv error: recv got EOF from 127.0.97.2:54073 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 26 21:37:48 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:48 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:48.839967 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:48.831634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:53853)
0926 21:37:48.832141 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:48.832158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:48.832276 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:48.832601 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:48.832604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:48.832982 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:48.833120 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:48.833128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.834179 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.834183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:48.835076 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:48.835087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.835720 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.835723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:48.835852 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:48.836540 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:48.836562 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:48.839704 (+ 3142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":288,"threads_started":1}
W20260926 21:37:48.840152 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:48.831768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53853 (local address 127.0.97.62:35755)
0926 21:37:48.832026 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:48.832033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:48.832270 (+ 237us) server_negotiation.cc:407] Connection header received
0926 21:37:48.832400 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:48.832404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:48.832468 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:48.832554 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:48.833310 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.834061 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.835196 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.835595 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.835687 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:48.839902 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53853: BlockingRecv error: recv got EOF from 127.0.97.3:53853 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 21:37:48 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:48 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:48.922924 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:48.914504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:38423)
0926 21:37:48.915024 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:48.915041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:48.915154 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:48.915452 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:48.915456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:48.915783 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:48.915937 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:48.915945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.916954 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.916958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:48.917922 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:48.917934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.918634 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.918638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:48.918773 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:48.919384 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:48.919403 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:48.922556 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":306,"threads_started":1}
W20260926 21:37:48.923023 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:48.914580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38423 (local address 127.0.97.62:35755)
0926 21:37:48.914801 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:48.914806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:48.915120 (+ 314us) server_negotiation.cc:407] Connection header received
0926 21:37:48.915257 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:48.915260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:48.915327 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:48.915415 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:48.916059 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.916804 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.918061 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:48.918489 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:48.918584 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:48.922767 (+ 4183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38423: BlockingRecv error: recv got EOF from 127.0.97.1:38423 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 21:37:49 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:49 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:49.381014 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:49.372611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:38151)
0926 21:37:49.373280 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:49.373301 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:49.373448 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:49.373809 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:49.373813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:49.374156 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:49.374317 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:49.374325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.375321 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.375324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:49.376196 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:49.376206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.376829 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.376833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:49.376971 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:49.377645 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:49.377663 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:49.380761 (+ 3098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":301,"threads_started":1}
W20260926 21:37:49.381165 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:49.372630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38151 (local address 127.0.97.62:35755)
0926 21:37:49.372886 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:49.372896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:49.373397 (+ 501us) server_negotiation.cc:407] Connection header received
0926 21:37:49.373611 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:49.373615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:49.373679 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:49.373755 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:49.374441 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.375197 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.376331 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.376706 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.376844 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:49.380954 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38151: BlockingRecv error: recv got EOF from 127.0.97.2:38151 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 21:37:49 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:49 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:49.849474 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:49.840864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:41079)
0926 21:37:49.841421 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:49.841436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:49.841596 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:49.841927 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:49.841932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:49.842293 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:49.842449 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:49.842457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.843446 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.843452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:49.844408 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:49.844420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.845112 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.845117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:49.845328 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:49.846061 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:49.846079 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:49.849158 (+ 3079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":313,"threads_started":1}
W20260926 21:37:49.849650 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:49.841047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41079 (local address 127.0.97.62:35755)
0926 21:37:49.841304 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:49.841311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:49.841523 (+ 212us) server_negotiation.cc:407] Connection header received
0926 21:37:49.841720 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:49.841725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:49.841791 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:49.841900 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:49.842592 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.843294 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.844561 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.844965 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.845101 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:49.849427 (+ 4326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41079: BlockingRecv error: recv got EOF from 127.0.97.3:41079 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 21:37:49 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:49 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:49.932947 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:49.924038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:35617)
0926 21:37:49.924592 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:49.924608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:49.924774 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:49.925121 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:49.925125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:49.925543 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:49.925692 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:49.925700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.926706 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.926714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:49.927681 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:49.927692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.928403 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.928408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:49.928600 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:49.929175 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:49.929242 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:49.932612 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":352,"threads_started":1}
W20260926 21:37:49.933065 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:49.924137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35617 (local address 127.0.97.62:35755)
0926 21:37:49.924358 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:49.924369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:49.924706 (+ 337us) server_negotiation.cc:407] Connection header received
0926 21:37:49.924912 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:49.924918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:49.924985 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:49.925095 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:49.925856 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.926576 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.927837 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:49.928252 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:49.928393 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:49.932832 (+ 4439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35617: BlockingRecv error: recv got EOF from 127.0.97.1:35617 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 21:37:50 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:50 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:50.390923 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:50.382086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:37823)
0926 21:37:50.382543 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:50.382558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:50.382675 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:50.382987 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:50.382990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:50.383318 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:50.383479 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:50.383488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.384884 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.384887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:50.385918 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:50.385931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.386966 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.386972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:50.387117 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:50.387797 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:50.387819 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:50.390705 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":261,"threads_started":1}
W20260926 21:37:50.391173 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:50.382235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37823 (local address 127.0.97.62:35755)
0926 21:37:50.382490 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:50.382497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:50.382662 (+ 165us) server_negotiation.cc:407] Connection header received
0926 21:37:50.382792 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:50.382797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:50.382861 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:50.382942 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:50.383621 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.384749 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.386088 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.386805 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.386925 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:50.390946 (+ 4021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37823: BlockingRecv error: recv got EOF from 127.0.97.2:37823 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 26 21:37:50 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:50 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:50.859153 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:50.850522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:57713)
0926 21:37:50.851036 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:50.851053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:50.851171 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:50.851541 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:50.851545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:50.851961 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:50.852115 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:50.852124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.853162 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.853166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:50.854222 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:50.854234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.855124 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.855129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:50.855285 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:50.855969 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:50.856030 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:50.858931 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":285,"threads_started":1}
W20260926 21:37:50.859431 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:50.850593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57713 (local address 127.0.97.62:35755)
0926 21:37:50.850789 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:50.850796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:50.851121 (+ 325us) server_negotiation.cc:407] Connection header received
0926 21:37:50.851292 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:50.851297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:50.851376 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:50.851456 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:50.852256 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.853036 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.854369 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.854966 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.855095 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:50.859127 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57713: BlockingRecv error: recv got EOF from 127.0.97.3:57713 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 21:37:50 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:50 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:50.942471 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:50.934168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:57583)
0926 21:37:50.934749 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:50.934769 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:50.934948 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:50.935365 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:50.935369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:50.935846 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:50.936012 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:50.936021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.937039 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.937043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:50.938015 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:50.938026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.938673 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.938678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:50.938797 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:50.939380 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:50.939396 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:50.942105 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":277,"threads_started":1}
W20260926 21:37:50.942512 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:50.934394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57583 (local address 127.0.97.62:35755)
0926 21:37:50.934604 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:50.934610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:50.934884 (+ 274us) server_negotiation.cc:407] Connection header received
0926 21:37:50.935094 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:50.935100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:50.935172 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:50.935346 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:50.936154 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.936906 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.938126 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:50.938559 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:50.938657 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:50.942295 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57583: BlockingRecv error: recv got EOF from 127.0.97.1:57583 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 21:37:51 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:51 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:51.400617 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:51.391867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:36815)
0926 21:37:51.392410 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:51.392426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:51.392573 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:51.392938 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:51.392943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:51.393498 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:51.393655 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:51.393663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.394684 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.394691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:51.395702 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:51.395714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.396469 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.396474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:51.396658 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:51.397272 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:51.397290 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:51.400266 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":246,"threads_started":1}
W20260926 21:37:51.400707 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:51.392085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36815 (local address 127.0.97.62:35755)
0926 21:37:51.392300 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:51.392307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:51.392508 (+ 201us) server_negotiation.cc:407] Connection header received
0926 21:37:51.392703 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:51.392708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:51.392801 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:51.392911 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:51.393803 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.394549 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.395852 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.396317 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.396458 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:51.400437 (+ 3979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36815: BlockingRecv error: recv got EOF from 127.0.97.2:36815 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 21:37:51 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:51 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:51.868158 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:51.860062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:39451)
0926 21:37:51.860528 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:51.860545 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:51.860654 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:51.860944 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:51.860947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:51.861442 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:51.861612 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:51.861620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.862621 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.862624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:51.863528 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:51.863539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.864221 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.864225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:51.864478 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:51.865139 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:51.865157 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:51.867907 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":226,"threads_started":1}
W20260926 21:37:51.868399 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:51.860211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39451 (local address 127.0.97.62:35755)
0926 21:37:51.860429 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:51.860435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:51.860625 (+ 190us) server_negotiation.cc:407] Connection header received
0926 21:37:51.860753 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:51.860757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:51.860816 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:51.860917 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:51.861742 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.862481 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.863685 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.864091 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.864188 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:51.868155 (+ 3967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39451: BlockingRecv error: recv got EOF from 127.0.97.3:39451 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 21:37:51 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:51 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:51.952061 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:51.943552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:37891)
0926 21:37:51.944134 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:51.944151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:51.944259 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:51.944840 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:51.944844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:51.945281 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:51.945443 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:51.945451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.946486 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.946490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:51.947455 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:51.947466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.948317 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.948322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:51.948438 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:51.949145 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:51.949162 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:51.951839 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":345,"threads_started":1}
W20260926 21:37:51.952361 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:51.943608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37891 (local address 127.0.97.62:35755)
0926 21:37:51.943839 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:51.943848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:51.944216 (+ 368us) server_negotiation.cc:407] Connection header received
0926 21:37:51.944588 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:51.944593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:51.944677 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:51.944801 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:51.945569 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.946366 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.947585 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:51.948133 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:51.948288 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:51.952042 (+ 3754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37891: BlockingRecv error: recv got EOF from 127.0.97.1:37891 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 21:37:52 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:52 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:52.410164 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:52.401791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:51455)
0926 21:37:52.402348 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:52.402365 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:52.402478 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:52.402771 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:52.402774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:52.403138 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:52.403291 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:52.403300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.404261 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.404264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:52.405165 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:52.405176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.406174 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.406180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:52.406344 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:52.407061 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:52.407078 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:52.409930 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":322,"threads_started":1}
W20260926 21:37:52.410344 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:52.401850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51455 (local address 127.0.97.62:35755)
0926 21:37:52.402041 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:52.402045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:52.402456 (+ 411us) server_negotiation.cc:407] Connection header received
0926 21:37:52.402580 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:52.402583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:52.402645 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:52.402735 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:52.403440 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.404141 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.405394 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.405994 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.406160 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:52.410157 (+ 3997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51455: BlockingRecv error: recv got EOF from 127.0.97.2:51455 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 21:37:52 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:52 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:52.883602 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:52.869043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:34603)
0926 21:37:52.869591 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:52.869608 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:52.869714 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:52.869999 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:52.870003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:52.870347 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:52.870533 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:52.870541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.871492 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.871496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:52.872375 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:52.872385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.873051 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.873055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:52.873169 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:52.880354 (+ 7185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:52.880373 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:52.883256 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":310,"threads_started":1}
W20260926 21:37:52.883728 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:52.869243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34603 (local address 127.0.97.62:35755)
0926 21:37:52.869419 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:52.869423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:52.869689 (+ 266us) server_negotiation.cc:407] Connection header received
0926 21:37:52.869813 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:52.869817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:52.869875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:52.869954 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:52.870653 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.871380 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.872508 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.872918 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.873018 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:52.883486 (+ 10468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34603: BlockingRecv error: recv got EOF from 127.0.97.3:34603 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 21:37:52 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:52 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:52.961948 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:52.953107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:57009)
0926 21:37:52.953795 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:52.953812 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:52.953979 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:52.954324 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:52.954329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:52.954777 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:52.954940 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:52.954950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.955967 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.955972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:52.956983 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:52.956995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.957941 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.957947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:52.958116 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:52.958764 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:52.958782 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:52.961585 (+ 2803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":286,"threads_started":1}
W20260926 21:37:52.962041 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:52.953291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57009 (local address 127.0.97.62:35755)
0926 21:37:52.953512 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:52.953519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:52.953914 (+ 395us) server_negotiation.cc:407] Connection header received
0926 21:37:52.954118 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:52.954124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:52.954192 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:52.954296 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:52.955093 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.955810 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.957132 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:52.957776 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:52.957929 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:52.961795 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57009: BlockingRecv error: recv got EOF from 127.0.97.1:57009 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 21:37:53 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:53 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:53.419284 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:53.411261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:49465)
0926 21:37:53.411759 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:53.411775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:53.411894 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:53.412214 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:53.412217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:53.412642 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:53.412811 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:53.412822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.413817 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.413823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:53.414776 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:53.414787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.415453 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.415457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:53.415567 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:53.416168 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:53.416186 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:53.419042 (+ 2856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":223,"threads_started":1}
W20260926 21:37:53.419647 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:53.411331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49465 (local address 127.0.97.62:35755)
0926 21:37:53.411591 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:53.411598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:53.411859 (+ 261us) server_negotiation.cc:407] Connection header received
0926 21:37:53.412021 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:53.412024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:53.412084 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:53.412163 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:53.412935 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.413661 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.414906 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.415286 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.415373 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:53.419295 (+ 3922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49465: BlockingRecv error: recv got EOF from 127.0.97.2:49465 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 26 21:37:53 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:53 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:53.893373 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:53.884662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:56813)
0926 21:37:53.885353 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:53.885373 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:53.885509 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:53.885868 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:53.885871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:53.886279 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:53.886439 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:53.886447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.887465 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.887469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:53.888373 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:53.888385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.889386 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.889394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:53.889525 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:53.890234 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:53.890252 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:53.893056 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":299,"threads_started":1}
W20260926 21:37:53.893636 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:53.884824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56813 (local address 127.0.97.62:35755)
0926 21:37:53.885022 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:53.885027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:53.885487 (+ 460us) server_negotiation.cc:407] Connection header received
0926 21:37:53.885639 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:53.885644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:53.885735 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:53.885826 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:53.886580 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.887343 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.888508 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.889178 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.889325 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:53.893377 (+ 4052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56813: BlockingRecv error: recv got EOF from 127.0.97.3:56813 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 21:37:53 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:53 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:53.971760 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:53.963057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:36869)
0926 21:37:53.963639 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:53.963656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:53.963825 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:53.964191 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:53.964196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:53.964603 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:53.964745 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:53.964753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.965862 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.965869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:53.966790 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:53.966801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.967569 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.967575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:53.967742 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:53.968459 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:53.968477 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:53.971379 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":331,"threads_started":1}
W20260926 21:37:53.971855 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:53.963206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36869 (local address 127.0.97.62:35755)
0926 21:37:53.963413 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:53.963419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:53.963753 (+ 334us) server_negotiation.cc:407] Connection header received
0926 21:37:53.963961 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:53.963967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:53.964045 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:53.964156 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:53.964902 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.965706 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.966942 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:53.967381 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:53.967527 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:53.971580 (+ 4053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36869: BlockingRecv error: recv got EOF from 127.0.97.1:36869 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 21:37:54 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:54 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:54.429109 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:54.420353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:57153)
0926 21:37:54.420892 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:54.420908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:54.421024 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:54.421361 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:54.421365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:54.421886 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:54.422087 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:54.422097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.423682 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.423687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:54.424598 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:54.424609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.425373 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.425380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:54.425533 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:54.426117 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:54.426134 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:54.428877 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":317,"threads_started":1}
W20260926 21:37:54.429323 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:54.420496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57153 (local address 127.0.97.62:35755)
0926 21:37:54.420672 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:54.420676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:54.420987 (+ 311us) server_negotiation.cc:407] Connection header received
0926 21:37:54.421129 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:54.421133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:54.421216 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:54.421310 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:54.422278 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.423527 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.424747 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.425166 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.425332 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:54.429081 (+ 3749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57153: BlockingRecv error: recv got EOF from 127.0.97.2:57153 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 21:37:54 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:54 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:54.903002 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:54.894512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:60599)
0926 21:37:54.895072 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:54.895088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:54.895228 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:54.895539 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:54.895542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:54.895990 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:54.896134 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:54.896143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.897596 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.897600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:54.898669 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:54.898680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.899325 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.899329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:54.899437 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:54.900064 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:54.900081 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:54.902778 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":306,"threads_started":1}
W20260926 21:37:54.903261 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:54.894577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60599 (local address 127.0.97.62:35755)
0926 21:37:54.894776 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:54.894781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:54.895167 (+ 386us) server_negotiation.cc:407] Connection header received
0926 21:37:54.895315 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:54.895319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:54.895379 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:54.895463 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:54.896287 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.897404 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.898791 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.899180 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.899272 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:54.902961 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60599: BlockingRecv error: recv got EOF from 127.0.97.3:60599 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 21:37:54 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:54 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:54.981027 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:54.972829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:36981)
0926 21:37:54.973421 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:54.973438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:54.973548 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:54.973840 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:54.973844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:54.974221 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:54.974373 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:54.974381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.975313 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.975316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:54.976237 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:54.976248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.976922 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.976927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:54.977064 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:54.977649 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:54.977666 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:54.980676 (+ 3010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":325,"threads_started":1}
W20260926 21:37:54.981096 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:54.972873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36981 (local address 127.0.97.62:35755)
0926 21:37:54.973097 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:54.973102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:54.973520 (+ 418us) server_negotiation.cc:407] Connection header received
0926 21:37:54.973655 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:54.973658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:54.973717 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:54.973798 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:54.974488 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.975206 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.976355 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:54.976795 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:54.976898 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:54.980864 (+ 3966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36981: BlockingRecv error: recv got EOF from 127.0.97.1:36981 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 21:37:55 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:55 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:55.439153 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:55.430213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:50919)
0926 21:37:55.430778 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:55.430794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:55.430974 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:55.431393 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:55.431401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:55.431911 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:55.432164 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:55.432182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.433361 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.433365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:55.434547 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:55.434561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.435231 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.435236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:55.435361 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:55.435967 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:55.435987 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:55.438910 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":338,"threads_started":1}
W20260926 21:37:55.439350 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:55.430295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50919 (local address 127.0.97.62:35755)
0926 21:37:55.430570 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:55.430578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:55.430927 (+ 349us) server_negotiation.cc:407] Connection header received
0926 21:37:55.431116 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:55.431124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:55.431217 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:55.431336 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:55.432327 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.433242 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.434685 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.435080 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.435170 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:55.439119 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50919: BlockingRecv error: recv got EOF from 127.0.97.2:50919 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 21:37:55 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:55 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:55.912240 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:55.904195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:48335)
0926 21:37:55.904666 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:55.904682 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:55.904797 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:55.905089 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:55.905093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:55.905548 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:55.905706 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:55.905714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.906849 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.906853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:55.907757 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:55.907767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.908517 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.908523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:55.908722 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:55.909323 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:55.909339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:55.911970 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":259,"threads_started":1}
W20260926 21:37:55.912530 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:55.904345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48335 (local address 127.0.97.62:35755)
0926 21:37:55.904583 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:55.904589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:55.904769 (+ 180us) server_negotiation.cc:407] Connection header received
0926 21:37:55.904901 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:55.904904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:55.904964 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:55.905057 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:55.905832 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.906743 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.907914 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.908339 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.908479 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:55.912189 (+ 3710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48335: BlockingRecv error: recv got EOF from 127.0.97.3:48335 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 26 21:37:55 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:55 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:55.990357 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:55.982090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:42143)
0926 21:37:55.982523 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:55.982540 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:55.982657 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:55.982947 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:55.982950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:55.983307 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:55.983443 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:55.983451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.984567 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.984570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:55.985568 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:55.985579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.986335 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.986341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:55.986503 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:55.987170 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:55.987188 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:55.990024 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":252,"threads_started":1}
W20260926 21:37:55.990504 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:55.982148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42143 (local address 127.0.97.62:35755)
0926 21:37:55.982388 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:55.982395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:55.982629 (+ 234us) server_negotiation.cc:407] Connection header received
0926 21:37:55.982760 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:55.982764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:55.982823 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:55.982903 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:55.983687 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.984433 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.985727 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:55.986173 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:55.986315 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:55.990238 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42143: BlockingRecv error: recv got EOF from 127.0.97.1:42143 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 21:37:56 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:56 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:56.449023 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:56.440177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:56577)
0926 21:37:56.440671 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:56.440688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:56.440806 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:56.441160 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:56.441163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:56.441539 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:56.441681 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:56.441690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.442671 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.442675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:56.443687 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:56.443699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.444545 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.444550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:56.444681 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:56.445364 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:56.445386 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:56.448638 (+ 3252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":291,"threads_started":1}
W20260926 21:37:56.449126 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:56.440294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56577 (local address 127.0.97.62:35755)
0926 21:37:56.440525 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:56.440531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:56.440774 (+ 243us) server_negotiation.cc:407] Connection header received
0926 21:37:56.440934 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:56.440937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:56.440999 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:56.441088 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:56.441804 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.442535 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.443931 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.444354 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.444460 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:56.448856 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56577: BlockingRecv error: recv got EOF from 127.0.97.2:56577 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 21:37:56 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:56 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:56.921311 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:56.913240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:60005)
0926 21:37:56.913794 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:56.913810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:56.913920 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:56.914222 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:56.914225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:56.914579 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:56.914717 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:56.914724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.915733 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.915736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:56.916660 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:56.916672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.917378 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.917383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:56.917492 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:56.918119 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:56.918136 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:56.921017 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":334,"threads_started":1}
W20260926 21:37:56.921592 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:56.913459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60005 (local address 127.0.97.62:35755)
0926 21:37:56.913630 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:56.913634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:56.913886 (+ 252us) server_negotiation.cc:407] Connection header received
0926 21:37:56.914015 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:56.914018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:56.914097 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:56.914185 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:56.914855 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.915609 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.916791 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.917254 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.917349 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:56.921322 (+ 3973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60005: BlockingRecv error: recv got EOF from 127.0.97.3:60005 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 21:37:56 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:56 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:56.999737 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:56.991311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:50373)
0926 21:37:56.991907 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:56.991926 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:56.992078 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:56.992399 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:56.992404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:56.992902 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:56.993107 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:56.993116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.994139 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.994147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:56.995215 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:56.995226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.995974 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.995980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:56.996158 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:56.996717 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:56.996734 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:56.999368 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":338,"threads_started":1}
W20260926 21:37:56.999770 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:56.991424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50373 (local address 127.0.97.62:35755)
0926 21:37:56.991632 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:56.991640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:56.992031 (+ 391us) server_negotiation.cc:407] Connection header received
0926 21:37:56.992216 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:56.992220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:56.992278 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:56.992374 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:56.993267 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.994007 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.995355 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:56.995831 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:56.995961 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:56.999552 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50373: BlockingRecv error: recv got EOF from 127.0.97.1:50373 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 21:37:57 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:57 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:57.458743 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:57.450056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:51807)
0926 21:37:57.450598 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:57.450616 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:57.450732 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:57.451024 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:57.451029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:57.451411 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:57.451603 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:57.451612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:57.452588 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:57.452591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:57.453664 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:57.453678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:57.454588 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:57.454594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:57.454797 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:57.455505 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:57.455530 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:57.458500 (+ 2970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":306,"threads_started":1}
W20260926 21:37:57.458896 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:57.450107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51807 (local address 127.0.97.62:35755)
0926 21:37:57.450273 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:57.450278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:57.450692 (+ 414us) server_negotiation.cc:407] Connection header received
0926 21:37:57.450830 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:57.450834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:57.450896 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:57.450984 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:57.451727 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:57.452438 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:57.453797 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:57.454421 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:57.454546 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:57.458690 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51807: BlockingRecv error: recv got EOF from 127.0.97.2:51807 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 21:37:57 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:57 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:57.935722 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:57.922210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:58019)
0926 21:37:57.922677 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:57.922694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:57.922834 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:57.923175 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:57.923180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:57.923602 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:57.923799 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:57.923808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:57.924818 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:57.924824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:57.925810 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:57.925823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:57.926548 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:57.926553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:57.926717 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:57.932497 (+ 5780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:57.932517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:57.935368 (+ 2851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
W20260926 21:37:57.935842 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:57.922313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58019 (local address 127.0.97.62:35755)
0926 21:37:57.922543 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:57.922549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:57.922804 (+ 255us) server_negotiation.cc:407] Connection header received
0926 21:37:57.922939 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:57.922943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:57.923006 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:57.923143 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:57.923954 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:57.924685 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:57.925957 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:57.926361 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:57.926498 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:57.935609 (+ 9111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58019: BlockingRecv error: recv got EOF from 127.0.97.3:58019 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 21:37:58 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:58 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:58.009651 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:58.000848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:36819)
0926 21:37:58.001381 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:58.001397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:58.001522 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:58.001832 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:58.001836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:58.002209 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:58.002420 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:58.002432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.003747 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.003751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:58.004920 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:58.004933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.005724 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.005729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:58.005855 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:58.006512 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:58.006530 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:58.009415 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":282,"threads_started":1}
W20260926 21:37:58.009907 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:58.000898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36819 (local address 127.0.97.62:35755)
0926 21:37:58.001163 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:58.001170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:58.001489 (+ 319us) server_negotiation.cc:407] Connection header received
0926 21:37:58.001636 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:58.001641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:58.001705 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:58.001793 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:58.002615 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.003589 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.005055 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.005606 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.005703 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:58.009629 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36819: BlockingRecv error: recv got EOF from 127.0.97.1:36819 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 21:37:58 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:58 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:58.468263 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:58.459826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:35303)
0926 21:37:58.460414 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:58.460431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:58.460553 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:58.460942 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:58.460945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:58.461395 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:58.461561 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:58.461570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.462563 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.462567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:58.463458 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:58.463467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.464335 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.464343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:58.464504 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:58.465287 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:58.465305 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:58.468025 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":341,"threads_started":1}
W20260926 21:37:58.468516 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:58.459864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35303 (local address 127.0.97.62:35755)
0926 21:37:58.460059 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:58.460063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:58.460516 (+ 453us) server_negotiation.cc:407] Connection header received
0926 21:37:58.460711 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:58.460716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:58.460782 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:58.460877 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:58.461728 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.462429 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.463578 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.464177 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.464305 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:58.468246 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35303: BlockingRecv error: recv got EOF from 127.0.97.2:35303 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 21:37:58 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:58 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:58.945637 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:58.936772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:41297)
0926 21:37:58.937332 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:58.937356 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:58.937478 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:58.937813 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:58.937816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:58.938216 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:58.938367 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:58.938375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.939451 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.939454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:58.940648 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:58.940660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.941688 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.941693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:58.941820 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:58.942406 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:58.942423 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:58.945410 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":324,"threads_started":1}
W20260926 21:37:58.945987 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:58.936865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41297 (local address 127.0.97.62:35755)
0926 21:37:58.937050 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:58.937055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:58.937441 (+ 386us) server_negotiation.cc:407] Connection header received
0926 21:37:58.937614 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:58.937618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:58.937682 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:58.937773 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:58.938498 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.939295 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.940790 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:58.941538 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:58.941676 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:58.945708 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41297: BlockingRecv error: recv got EOF from 127.0.97.3:41297 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 21:37:59 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:59 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:59.019181 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:59.010694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:56203)
0926 21:37:59.011263 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:59.011279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:59.011404 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:59.011768 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:59.011772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:59.012184 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:59.012345 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:59.012354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.013433 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.013439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:59.014491 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:59.014507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.015460 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.015465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:59.015586 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:59.016132 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:59.016149 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:59.018956 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":313,"threads_started":1}
W20260926 21:37:59.019351 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:59.010784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56203 (local address 127.0.97.62:35755)
0926 21:37:59.011050 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:59.011054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:59.011398 (+ 344us) server_negotiation.cc:407] Connection header received
0926 21:37:59.011518 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:59.011522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:59.011585 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:59.011687 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:59.012485 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.013294 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.014654 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.015278 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.015400 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:59.019142 (+ 3742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56203: BlockingRecv error: recv got EOF from 127.0.97.1:56203 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 26 21:37:59 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:59 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:59.478171 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:59.469295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:54241)
0926 21:37:59.469865 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:59.469881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:59.470053 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:59.470401 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:59.470405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:59.470836 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:59.471004 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:59.471013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.472053 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.472059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:59.473002 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:59.473013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.473893 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.473901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:59.474083 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:59.474977 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:59.475002 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:59.477930 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":322,"threads_started":1}
W20260926 21:37:59.478351 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:59.469509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54241 (local address 127.0.97.62:35755)
0926 21:37:59.469742 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:59.469748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:59.469967 (+ 219us) server_negotiation.cc:407] Connection header received
0926 21:37:59.470190 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:59.470197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:59.470266 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:59.470369 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:59.471144 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.471888 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.473138 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.473692 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.473815 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:59.478108 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54241: BlockingRecv error: recv got EOF from 127.0.97.2:54241 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 21:37:59 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:37:59 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:37:59.955042 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:37:59.946583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:51585)
0926 21:37:59.947228 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 21:37:59.947243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:37:59.947402 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:37:59.947751 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:37:59.947756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:37:59.948274 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:37:59.948469 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:59.948478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.949556 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.949570 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:59.950542 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:37:59.950553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.951248 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.951253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:37:59.951423 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:59.952017 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:37:59.952034 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:37:59.954793 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":286,"threads_started":1}
W20260926 21:37:59.955243 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:37:59.946858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51585 (local address 127.0.97.62:35755)
0926 21:37:59.947111 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 21:37:59.947117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:37:59.947326 (+ 209us) server_negotiation.cc:407] Connection header received
0926 21:37:59.947543 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:37:59.947547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:37:59.947613 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:37:59.947718 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:37:59.948615 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.949424 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.950687 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:37:59.951107 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:37:59.951239 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:37:59.954987 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51585: BlockingRecv error: recv got EOF from 127.0.97.3:51585 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 21:38:00 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:38:00 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:38:00.028594 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:00.020163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:54989)
0926 21:38:00.020648 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:00.020668 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:00.020819 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:00.021232 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:00.021239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:00.021710 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:00.021884 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:00.021892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.023000 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.023006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:00.023943 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:00.023955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.024648 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.024654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:00.024829 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:00.025417 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:00.025434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:00.028228 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":453,"threads_started":1}
W20260926 21:38:00.028851 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:38:00.020287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54989 (local address 127.0.97.62:35755)
0926 21:38:00.020617 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:00.020624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:00.020771 (+ 147us) server_negotiation.cc:407] Connection header received
0926 21:38:00.021006 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:00.021010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:00.021071 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:00.021177 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:00.022028 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.022860 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.024089 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.024497 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.024633 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:00.028433 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54989: BlockingRecv error: recv got EOF from 127.0.97.1:54989 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Sep 26 21:38:00 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:38:00 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:38:00.487088 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:00.479151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:60077)
0926 21:38:00.479591 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:00.479608 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:00.479724 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:00.480009 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:00.480011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:00.480375 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:00.480573 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:00.480582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.481635 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.481640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:00.482559 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:00.482570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.483240 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.483244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:00.483356 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:00.484103 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:00.484123 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:00.486847 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":241,"threads_started":1}
W20260926 21:38:00.487246 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:38:00.479256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60077 (local address 127.0.97.62:35755)
0926 21:38:00.479514 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:00.479521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:00.479693 (+ 172us) server_negotiation.cc:407] Connection header received
0926 21:38:00.479825 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:00.479828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:00.479886 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:00.479969 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:00.480706 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.481495 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.482708 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.483117 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.483253 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:00.487034 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60077: BlockingRecv error: recv got EOF from 127.0.97.2:60077 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 21:38:00 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:38:00 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:38:00.964437 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:00.956090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.3:48213)
0926 21:38:00.956669 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:00.956689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:00.956856 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:00.957304 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:00.957308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:00.957753 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:00.957993 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:00.958007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.959057 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.959063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:00.959971 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:00.959981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.960700 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.960705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:00.960885 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:00.961588 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:00.961605 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:00.964092 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":378,"threads_started":1}
W20260926 21:38:00.964550 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:38:00.956294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48213 (local address 127.0.97.62:35755)
0926 21:38:00.956485 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:00.956492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:00.956785 (+ 293us) server_negotiation.cc:407] Connection header received
0926 21:38:00.957042 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:00.957048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:00.957138 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:00.957280 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:00.958200 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.958920 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.960114 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:00.960559 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:00.960691 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:00.964320 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48213: BlockingRecv error: recv got EOF from 127.0.97.3:48213 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 21:38:01 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:38:01 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:38:01.037757 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:01.029963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.1:45283)
0926 21:38:01.030473 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:01.030488 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:01.030621 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:01.031025 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:01.031030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:01.031459 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:01.031627 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:01.031635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.032564 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.032567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:01.033562 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:01.033573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.034229 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.034233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:01.034338 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:01.034888 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:01.034904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:01.037513 (+ 2609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":288,"threads_started":1}
W20260926 21:38:01.037919 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:38:01.030011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45283 (local address 127.0.97.62:35755)
0926 21:38:01.030199 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:01.030205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:01.030571 (+ 366us) server_negotiation.cc:407] Connection header received
0926 21:38:01.030753 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:01.030757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:01.030859 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:01.030957 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:01.031754 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.032453 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.033684 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.034106 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.034203 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:01.037692 (+ 3489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45283: BlockingRecv error: recv got EOF from 127.0.97.1:45283 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 21:38:01 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 21:38:01 dist-test-slave-lhhq krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 21:38:01.496462 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:01.488265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35755 (local address 127.0.97.2:56687)
0926 21:38:01.488862 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:01.488882 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:01.489014 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:01.489396 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:01.489399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:01.489795 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:01.489955 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:01.489964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.490912 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.490916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:01.491827 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:01.491838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.492516 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.492521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:01.492668 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:01.493323 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:01.493340 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:01.496158 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35755: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":381,"threads_started":1}
W20260926 21:38:01.496622 9351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:38:01.488310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56687 (local address 127.0.97.62:35755)
0926 21:38:01.488531 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:01.488540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:01.488985 (+ 445us) server_negotiation.cc:407] Connection header received
0926 21:38:01.489159 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:01.489164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:01.489271 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:01.489348 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:01.490073 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.490791 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.491942 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.492370 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.492461 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:01.496386 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56687: BlockingRecv error: recv got EOF from 127.0.97.2:56687 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:38:01.792075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9202
I20260926 21:38:01.805284 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:01.793408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.0.97.1:41535)
0926 21:38:01.793955 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:01.793962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:01.793991 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:38:01.794050 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:01.794056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:01.794298 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:01.794478 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:01.796133 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.797344 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.798612 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.799192 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.799370 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:01.801946 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:01.801955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:01.801970 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:01.802013 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:01.804523 (+ 2510us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 21:38:01.804912 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55460: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":270,"threads_started":1}
W20260926 21:38:01.805413 9662 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55460: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 21:38:01.805403 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9202: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:41535: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 21:38:01.805634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 9202
I20260926 21:38:01.822798 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9353
I20260926 21:38:01.834857 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:01.823874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.0.97.2:38897)
0926 21:38:01.824476 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:01.824484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:01.824514 (+ 30us) server_negotiation.cc:407] Connection header received
0926 21:38:01.824572 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:01.824579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:01.824810 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:01.825010 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:01.826369 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.827407 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.828615 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.829279 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.829412 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:01.831862 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:01.831872 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:01.831886 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:01.831921 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:01.834198 (+ 2277us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 21:38:01.834503 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56080: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":317,"threads_started":1}
W20260926 21:38:01.834896 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9353: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:38897: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 21:38:01.834972 9663 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56080: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 21:38:01.835076 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 9353
I20260926 21:38:01.851624 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9484
I20260926 21:38:01.863024 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:01.852593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.97.3:46669)
0926 21:38:01.853255 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:01.853270 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:01.853297 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:38:01.853357 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:01.853363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:01.853602 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:01.853762 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:01.854960 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.856026 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.857168 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:01.857754 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:01.857859 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:01.860140 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:01.860149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:01.860163 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:01.860214 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:01.862530 (+ 2316us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 21:38:01.862787 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54246: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":303,"threads_started":1}
W20260926 21:38:01.863158 9664 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54246: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 21:38:01.863197 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9484: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:46669: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 21:38:01.863333 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 9484
I20260926 21:38:01.879478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9133
I20260926 21:38:01.881028 9195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:01.994602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 9133
2026-09-26T21:38:02Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21090 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T21:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:38:02Z Disabled control of system clock
I20260926 21:38:02.071012 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38673
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 21:38:02.403601 9669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:02.403901 9669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:02.404047 9669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:02.411599 9669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:38:02.411713 9669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:02.411751 9669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:02.411824 9669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:38:02.411885 9669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:38:02.421370 9669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:02.422712 9669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:02.424566 9669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:02.436693 9677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:02.436667 9675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:02.437071 9674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:02.437534 9669 server_base.cc:1034] running on GCE node
I20260926 21:38:02.438409 9669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:02.439934 9669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:02.441220 9669 hybrid_clock.cc:648] HybridClock initialized: now 1790458682441134 us; error 44 us; skew 500 ppm
I20260926 21:38:02.441588 9669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:38:02.443984 9669 webserver.cc:460] Webserver started at http://127.0.97.62:38547/ using document root <none> and password file <none>
I20260926 21:38:02.444720 9669 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:02.444849 9669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:02.445233 9669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:02.448113 9669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "708a8076b9b046aebb03f0b53d061f7a"
format_stamp: "Formatted at 2026-09-26 21:38:02 on dist-test-slave-lhhq"
server_key: "a366bb96aad06b28b771e4c742328462"
server_key_iv: "9f3d85da533bd98387eb91cc08b78349"
server_key_version: "encryptionkey@0"
I20260926 21:38:02.448958 9669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "708a8076b9b046aebb03f0b53d061f7a"
format_stamp: "Formatted at 2026-09-26 21:38:02 on dist-test-slave-lhhq"
server_key: "a366bb96aad06b28b771e4c742328462"
server_key_iv: "9f3d85da533bd98387eb91cc08b78349"
server_key_version: "encryptionkey@0"
I20260926 21:38:02.454198 9669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 21:38:02.458087 9683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:02.459481 9669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:38:02.459667 9669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "708a8076b9b046aebb03f0b53d061f7a"
format_stamp: "Formatted at 2026-09-26 21:38:02 on dist-test-slave-lhhq"
server_key: "a366bb96aad06b28b771e4c742328462"
server_key_iv: "9f3d85da533bd98387eb91cc08b78349"
server_key_version: "encryptionkey@0"
I20260926 21:38:02.459893 9669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:02.493767 9669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:02.494676 9669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:02.494968 9669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:02.513350 9669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38673
I20260926 21:38:02.513389 9734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38673 every 8 connection(s)
I20260926 21:38:02.514722 9669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:38:02.518548 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:02.523005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 9669
I20260926 21:38:02.523232 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:38:02.523983 388 external_mini_cluster.cc:1428] Setting key 894c91bc80fa41029d5bceed6818ae48
I20260926 21:38:02.530215 9735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a: Bootstrap starting.
I20260926 21:38:02.533484 9735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:02.534559 9735 log.cc:824] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:38:02.537783 9735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a: No bootstrap required, opened a new log
I20260926 21:38:02.538146 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:02.526246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.0.97.62:38673)
0926 21:38:02.527258 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:02.527269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:02.527310 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:38:02.529014 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:02.529038 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:02.529575 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:02.529984 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:02.530560 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:02.531939 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:02.533346 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:02.534113 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:02.534315 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:02.535565 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:02.535578 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:02.535602 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:02.536279 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:02.536360 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:02.536380 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:02.536469 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:02.536731 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 21:38:02.537079 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":353,"threads_started":1}
I20260926 21:38:02.545688 9735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a8076b9b046aebb03f0b53d061f7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38673 } }
I20260926 21:38:02.546096 9735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:02.546247 9735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708a8076b9b046aebb03f0b53d061f7a, State: Initialized, Role: FOLLOWER
I20260926 21:38:02.546878 9735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a8076b9b046aebb03f0b53d061f7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38673 } }
I20260926 21:38:02.547116 9735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:38:02.547261 9735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:38:02.547439 9735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:02.549676 9735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a8076b9b046aebb03f0b53d061f7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38673 } }
I20260926 21:38:02.550179 9735 leader_election.cc:302] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 708a8076b9b046aebb03f0b53d061f7a; no voters:
I20260926 21:38:02.550812 9735 leader_election.cc:288] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:38:02.550945 9740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:38:02.551687 9740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [term 1 LEADER]: Becoming Leader. State: Replica: 708a8076b9b046aebb03f0b53d061f7a, State: Running, Role: LEADER
I20260926 21:38:02.552263 9740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a8076b9b046aebb03f0b53d061f7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38673 } }
I20260926 21:38:02.553046 9735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:38:02.555368 9742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 708a8076b9b046aebb03f0b53d061f7a. Latest consensus state: current_term: 1 leader_uuid: "708a8076b9b046aebb03f0b53d061f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a8076b9b046aebb03f0b53d061f7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38673 } } }
I20260926 21:38:02.555856 9742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [sys.catalog]: This master's current role is: LEADER
I20260926 21:38:02.556469 9741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "708a8076b9b046aebb03f0b53d061f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a8076b9b046aebb03f0b53d061f7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38673 } } }
I20260926 21:38:02.556844 9741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a [sys.catalog]: This master's current role is: LEADER
I20260926 21:38:02.558125 9749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:38:02.562486 9749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:38:02.571144 9749 catalog_manager.cc:1347] Generated new cluster ID: 8930fddcaabb4486a4abb164105d74e3
I20260926 21:38:02.571266 9749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:38:02.594103 9749 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:38:02.595520 9749 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:38:02.613442 9749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 708a8076b9b046aebb03f0b53d061f7a: Generated new TSK 0
I20260926 21:38:02.614364 9749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:38:02.627655 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38673
--builtin_ntp_servers=127.0.97.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 21:38:02.964751 9759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:02.965044 9759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:02.965245 9759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:02.972579 9759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:02.972692 9759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:02.972867 9759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:38:02.982445 9759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:02.983807 9759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:02.985708 9759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:02.999321 9764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:02.999326 9765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:02.999377 9767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:03.000188 9759 server_base.cc:1034] running on GCE node
I20260926 21:38:03.000782 9759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:03.001797 9759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:03.003019 9759 hybrid_clock.cc:648] HybridClock initialized: now 1790458683002965 us; error 40 us; skew 500 ppm
I20260926 21:38:03.003356 9759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:38:03.011865 9759 webserver.cc:460] Webserver started at http://127.0.97.1:43715/ using document root <none> and password file <none>
I20260926 21:38:03.012749 9759 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:03.012882 9759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:03.013310 9759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:03.016316 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "163d856189514adbba7e6af6efc50949"
format_stamp: "Formatted at 2026-09-26 21:38:03 on dist-test-slave-lhhq"
server_key: "b6edd49723efde53c398fe1370884244"
server_key_iv: "4a4bc78935dfa44b14e9281d78b0fd8d"
server_key_version: "encryptionkey@0"
I20260926 21:38:03.017155 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "163d856189514adbba7e6af6efc50949"
format_stamp: "Formatted at 2026-09-26 21:38:03 on dist-test-slave-lhhq"
server_key: "b6edd49723efde53c398fe1370884244"
server_key_iv: "4a4bc78935dfa44b14e9281d78b0fd8d"
server_key_version: "encryptionkey@0"
I20260926 21:38:03.022604 9759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:38:03.026465 9773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:03.027904 9759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:38:03.028105 9759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "163d856189514adbba7e6af6efc50949"
format_stamp: "Formatted at 2026-09-26 21:38:03 on dist-test-slave-lhhq"
server_key: "b6edd49723efde53c398fe1370884244"
server_key_iv: "4a4bc78935dfa44b14e9281d78b0fd8d"
server_key_version: "encryptionkey@0"
I20260926 21:38:03.028357 9759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:03.060364 9759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:03.061347 9759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:03.061626 9759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:03.062594 9759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:03.064474 9759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:03.064575 9759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:03.064698 9759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:03.064772 9759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:03.098719 9759 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45443
I20260926 21:38:03.098810 9885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45443 every 8 connection(s)
I20260926 21:38:03.100067 9759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:38:03.103488 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 9759
I20260926 21:38:03.103703 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:38:03.104331 388 external_mini_cluster.cc:1428] Setting key 9cc7febd09c5f479e9b2d4395aa2686e
I20260926 21:38:03.107025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38673
--builtin_ntp_servers=127.0.97.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 21:38:03.113921 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:03.102810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33165 (local address 127.0.97.62:38673)
0926 21:38:03.103357 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:03.103364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:03.104915 (+ 1551us) server_negotiation.cc:407] Connection header received
0926 21:38:03.106947 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:03.106952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:03.107039 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:03.107170 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:03.108358 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:03.109122 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:03.110877 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:03.111291 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:03.111426 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:03.112955 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:03.112961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:03.112966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:03.112995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:03.113036 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:03.113048 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:03.113213 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:03.113451 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 21:38:03.113556 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":267,"threads_started":1}
I20260926 21:38:03.115067 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:03.103538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38673 (local address 127.0.97.1:33165)
0926 21:38:03.104725 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:03.104773 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:03.106655 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:03.107437 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:03.107446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:03.107550 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:03.108179 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:03.108198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:03.109340 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:03.109344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:03.110713 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:03.110728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:03.111544 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:03.111551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:03.111808 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:03.112581 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:38:03.112583 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:03.112643 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:38:03.112736 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:03.113206 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:03.113221 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:03.113231 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:38:03.113443 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 21:38:03.113914 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":385,"threads_started":1}
I20260926 21:38:03.116824 9886 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38673
I20260926 21:38:03.117146 9886 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:03.117959 9886 heartbeater.cc:511] Master 127.0.97.62:38673 requested a full tablet report, sending...
I20260926 21:38:03.120422 9700 ts_manager.cc:194] Registered new tserver with Master: 163d856189514adbba7e6af6efc50949 (127.0.97.1:45443)
I20260926 21:38:03.122786 9700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:33165
W20260926 21:38:03.440937 9890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:03.441306 9890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:03.441422 9890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:03.448722 9890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:03.448834 9890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:03.448938 9890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:38:03.458411 9890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:03.459692 9890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:03.461639 9890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:03.477770 9896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:03.477735 9898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:03.477810 9895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:03.478658 9890 server_base.cc:1034] running on GCE node
I20260926 21:38:03.479388 9890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:03.480397 9890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:03.481626 9890 hybrid_clock.cc:648] HybridClock initialized: now 1790458683481576 us; error 54 us; skew 500 ppm
I20260926 21:38:03.481969 9890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:38:03.484314 9890 webserver.cc:460] Webserver started at http://127.0.97.2:44989/ using document root <none> and password file <none>
I20260926 21:38:03.485054 9890 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:03.485226 9890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:03.485574 9890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:03.488530 9890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "0ae74f02c85b4a1083e8257ea38b9140"
format_stamp: "Formatted at 2026-09-26 21:38:03 on dist-test-slave-lhhq"
server_key: "c61a71a146ad1d79ae4fb51348456f67"
server_key_iv: "0ea3291effe40291ded26e60c5f1d8ae"
server_key_version: "encryptionkey@0"
I20260926 21:38:03.489452 9890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0ae74f02c85b4a1083e8257ea38b9140"
format_stamp: "Formatted at 2026-09-26 21:38:03 on dist-test-slave-lhhq"
server_key: "c61a71a146ad1d79ae4fb51348456f67"
server_key_iv: "0ea3291effe40291ded26e60c5f1d8ae"
server_key_version: "encryptionkey@0"
I20260926 21:38:03.494890 9890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:38:03.499011 9904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:03.500506 9890 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 21:38:03.500705 9890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "0ae74f02c85b4a1083e8257ea38b9140"
format_stamp: "Formatted at 2026-09-26 21:38:03 on dist-test-slave-lhhq"
server_key: "c61a71a146ad1d79ae4fb51348456f67"
server_key_iv: "0ea3291effe40291ded26e60c5f1d8ae"
server_key_version: "encryptionkey@0"
I20260926 21:38:03.500958 9890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:03.519802 9890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:03.520721 9890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:03.521008 9890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:03.522220 9890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:03.523751 9890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:03.523878 9890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:03.524017 9890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:03.524106 9890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:03.558274 9890 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41075
I20260926 21:38:03.558328 10016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41075 every 8 connection(s)
I20260926 21:38:03.559592 9890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:38:03.562304 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 9890
I20260926 21:38:03.562526 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:38:03.563031 388 external_mini_cluster.cc:1428] Setting key ec305b8b6c87375384659f39626f454d
I20260926 21:38:03.565953 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38673
--builtin_ntp_servers=127.0.97.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 21:38:03.572609 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:03.562234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39509 (local address 127.0.97.62:38673)
0926 21:38:03.562446 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:03.562452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:03.564377 (+ 1925us) server_negotiation.cc:407] Connection header received
0926 21:38:03.566269 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:03.566274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:03.566361 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:03.566463 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:03.567575 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:03.568290 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:03.569919 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:03.570321 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:03.570451 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:03.571818 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:03.571824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:03.571829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:03.571855 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:03.571883 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:03.571894 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:03.571983 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:03.572205 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 21:38:03.572301 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:38:03.573740 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:03.562982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38673 (local address 127.0.97.2:39509)
0926 21:38:03.564195 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:03.564245 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:03.565937 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:03.566738 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:03.566747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:03.566840 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:03.567409 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:03.567425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:03.568441 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:03.568451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:03.569772 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:03.569788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:03.570539 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:03.570546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:03.570785 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:03.571501 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:38:03.571504 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:03.571544 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:38:03.571622 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:03.571982 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:03.571994 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:03.572004 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:38:03.572192 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 21:38:03.572599 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":391,"threads_started":1}
I20260926 21:38:03.575245 10017 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38673
I20260926 21:38:03.575549 10017 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:03.576335 10017 heartbeater.cc:511] Master 127.0.97.62:38673 requested a full tablet report, sending...
I20260926 21:38:03.577883 9700 ts_manager.cc:194] Registered new tserver with Master: 0ae74f02c85b4a1083e8257ea38b9140 (127.0.97.2:41075)
I20260926 21:38:03.579101 9700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:39509
W20260926 21:38:03.900440 10020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:03.900736 10020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:03.900818 10020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:03.908212 10020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:03.908313 10020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:03.908417 10020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:38:03.917251 10020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:03.918550 10020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:03.920265 10020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:03.933439 10025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:03.933439 10028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:03.934016 10026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:03.934564 10020 server_base.cc:1034] running on GCE node
I20260926 21:38:03.935222 10020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:03.936556 10020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:03.937786 10020 hybrid_clock.cc:648] HybridClock initialized: now 1790458683937725 us; error 69 us; skew 500 ppm
I20260926 21:38:03.938144 10020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:38:03.940470 10020 webserver.cc:460] Webserver started at http://127.0.97.3:46165/ using document root <none> and password file <none>
I20260926 21:38:03.941263 10020 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:03.941394 10020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:03.941720 10020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:03.944685 10020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "033b4a59e501409a8cc340fb0865b923"
format_stamp: "Formatted at 2026-09-26 21:38:03 on dist-test-slave-lhhq"
server_key: "a0f87460c0d04df5e319aeb14f0dfe37"
server_key_iv: "5a910720b1056959a1413446ff654f32"
server_key_version: "encryptionkey@0"
I20260926 21:38:03.945530 10020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "033b4a59e501409a8cc340fb0865b923"
format_stamp: "Formatted at 2026-09-26 21:38:03 on dist-test-slave-lhhq"
server_key: "a0f87460c0d04df5e319aeb14f0dfe37"
server_key_iv: "5a910720b1056959a1413446ff654f32"
server_key_version: "encryptionkey@0"
I20260926 21:38:03.950686 10020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 21:38:03.954174 10034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:03.955549 10020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:38:03.955775 10020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "033b4a59e501409a8cc340fb0865b923"
format_stamp: "Formatted at 2026-09-26 21:38:03 on dist-test-slave-lhhq"
server_key: "a0f87460c0d04df5e319aeb14f0dfe37"
server_key_iv: "5a910720b1056959a1413446ff654f32"
server_key_version: "encryptionkey@0"
I20260926 21:38:03.956020 10020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:03.982823 10020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:03.983733 10020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:03.983975 10020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:03.984992 10020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:03.986471 10020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:03.986555 10020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:03.986639 10020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:03.986699 10020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:04.018301 10020 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46725
I20260926 21:38:04.018355 10146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46725 every 8 connection(s)
I20260926 21:38:04.019647 10020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:38:04.029861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 10020
I20260926 21:38:04.030053 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:38:04.030471 388 external_mini_cluster.cc:1428] Setting key 8ad25e4aeafa67dfc933849b6527d41d
I20260926 21:38:04.032809 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:04.022377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42083 (local address 127.0.97.62:38673)
0926 21:38:04.022556 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:04.022560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:04.023977 (+ 1417us) server_negotiation.cc:407] Connection header received
0926 21:38:04.025872 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:04.025877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:04.025972 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:04.026088 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:04.027280 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.028031 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.029756 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.030276 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.030406 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:04.031801 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:04.031807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:04.031811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:04.031837 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:04.031865 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:04.031876 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:04.031972 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:04.032288 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 21:38:04.032408 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:38:04.034034 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:04.022813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38673 (local address 127.0.97.3:42083)
0926 21:38:04.023800 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:04.023838 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:04.025590 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:04.026333 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:04.026341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:04.026438 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:04.027104 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:04.027121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.028197 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.028202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:04.029617 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:04.029630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.030502 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.030510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:04.030740 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:04.031473 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:38:04.031475 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:04.031516 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:38:04.031604 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:04.032024 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:04.032033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:04.032039 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:38:04.032243 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 21:38:04.032657 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":323,"threads_started":1}
I20260926 21:38:04.035321 10147 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38673
I20260926 21:38:04.035634 10147 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:04.036386 10147 heartbeater.cc:511] Master 127.0.97.62:38673 requested a full tablet report, sending...
I20260926 21:38:04.037813 9700 ts_manager.cc:194] Registered new tserver with Master: 033b4a59e501409a8cc340fb0865b923 (127.0.97.3:46725)
I20260926 21:38:04.038592 9700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:42083
I20260926 21:38:04.046149 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 21:38:04.052161 9888 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:38:04.050674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.0.97.62:38673)
0926 21:38:04.050858 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:04.050862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:04.051274 (+ 412us) server_negotiation.cc:407] Connection header received
0926 21:38:04.051392 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:04.051396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:04.051471 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:04.051556 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:04.051878 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60350: BlockingRecv error: recv got EOF from 127.0.0.1:60350 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:38:04.052986 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9759
I20260926 21:38:04.060909 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:04.054044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.0.97.1:45443)
0926 21:38:04.054527 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:04.054535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:04.054558 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:38:04.054609 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:04.054615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:04.054829 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:04.054955 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:04.055284 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.056400 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.057671 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.058234 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.058330 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:04.059356 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:04.059365 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:04.059378 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:04.060005 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:04.060061 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:04.060080 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:04.060251 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:04.060356 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 21:38:04.060517 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":255,"threads_started":1}
I20260926 21:38:04.062196 9881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:04.141590 9886 heartbeater.cc:503] Master 127.0.97.62:38673 was elected leader, sending a full tablet report...
I20260926 21:38:04.143287 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 9759
W20260926 21:38:04.160084 9686 connection.cc:535] server connection from 127.0.97.1:33165 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 21:38:04.160409 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9890
I20260926 21:38:04.168464 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:04.161354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.0.97.2:41075)
0926 21:38:04.161948 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:04.161955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:04.161982 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:38:04.162051 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:04.162057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:04.162271 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:04.162478 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:04.162877 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.163930 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.165170 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.165832 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.165971 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:04.166984 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:04.166993 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:04.167010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:04.167646 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:04.167692 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:04.167710 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:04.167833 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:04.167980 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 21:38:04.168123 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":331,"threads_started":1}
I20260926 21:38:04.169745 10012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:04.250483 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 9890
I20260926 21:38:04.266877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 10020
I20260926 21:38:04.275398 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:04.267844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.0.97.3:46725)
0926 21:38:04.268485 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:04.268492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:04.268523 (+ 31us) server_negotiation.cc:407] Connection header received
0926 21:38:04.268589 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:04.268596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:04.268826 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:04.268998 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:04.269493 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.270519 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.271789 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:04.272356 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:04.272489 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:04.273880 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:04.273888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:04.273902 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:04.274512 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:04.274559 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:04.274588 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:04.274746 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:04.274906 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 21:38:04.275042 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":347,"threads_started":1}
I20260926 21:38:04.276702 10142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:04.357823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 10020
I20260926 21:38:04.374331 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 9669
I20260926 21:38:04.375535 9730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:04.477852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 9669
2026-09-26T21:38:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2478 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T21:38:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:38:04Z Disabled control of system clock
I20260926 21:38:04.548977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46853
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 21:38:04.869822 10166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:04.870095 10166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:04.870177 10166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:04.877090 10166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:38:04.877260 10166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:04.877329 10166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:04.877388 10166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:38:04.877434 10166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:38:04.886442 10166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:04.887713 10166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:04.889493 10166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:04.901000 10171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:04.901072 10174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:04.901068 10172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:04.902024 10166 server_base.cc:1034] running on GCE node
I20260926 21:38:04.902848 10166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:04.904191 10166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:04.905409 10166 hybrid_clock.cc:648] HybridClock initialized: now 1790458684905370 us; error 54 us; skew 500 ppm
I20260926 21:38:04.905753 10166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:38:04.907938 10166 webserver.cc:460] Webserver started at http://127.0.97.62:35113/ using document root <none> and password file <none>
I20260926 21:38:04.908648 10166 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:04.908787 10166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:04.909106 10166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:04.911907 10166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "3215cae2aa6b482ea8673bc617b6b6ed"
format_stamp: "Formatted at 2026-09-26 21:38:04 on dist-test-slave-lhhq"
server_key: "bce07c09db9bd5fa7e7c282533c48459"
server_key_iv: "03e1363e8ddd527d73e9b70ab5f6f1a4"
server_key_version: "encryptionkey@0"
I20260926 21:38:04.912727 10166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "3215cae2aa6b482ea8673bc617b6b6ed"
format_stamp: "Formatted at 2026-09-26 21:38:04 on dist-test-slave-lhhq"
server_key: "bce07c09db9bd5fa7e7c282533c48459"
server_key_iv: "03e1363e8ddd527d73e9b70ab5f6f1a4"
server_key_version: "encryptionkey@0"
I20260926 21:38:04.917721 10166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:38:04.920981 10180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:04.922317 10166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:38:04.922499 10166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "3215cae2aa6b482ea8673bc617b6b6ed"
format_stamp: "Formatted at 2026-09-26 21:38:04 on dist-test-slave-lhhq"
server_key: "bce07c09db9bd5fa7e7c282533c48459"
server_key_iv: "03e1363e8ddd527d73e9b70ab5f6f1a4"
server_key_version: "encryptionkey@0"
I20260926 21:38:04.922731 10166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:04.928169 10166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:04.928931 10166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:04.929303 10166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:04.946763 10166 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46853
I20260926 21:38:04.946789 10231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46853 every 8 connection(s)
I20260926 21:38:04.948112 10166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:38:04.949436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 10166
I20260926 21:38:04.949643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:38:04.950455 388 external_mini_cluster.cc:1428] Setting key 96ca5623f1b1ffd05456020f19eeae73
I20260926 21:38:04.953043 10232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:04.958698 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:04.952497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.0.97.62:46853)
0926 21:38:04.953513 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:04.953525 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:04.953566 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:38:04.955098 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:04.955119 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:04.955536 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:04.955944 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:04.957015 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:04.957024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:04.957039 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:04.957062 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:04.957140 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:04.957163 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:04.957442 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:04.957598 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 21:38:04.957827 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":318,"threads_started":1}
I20260926 21:38:04.964145 10232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed: Bootstrap starting.
I20260926 21:38:04.967075 10232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:04.968082 10232 log.cc:824] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed: Log is configured to *not* fsync() on all Append() calls
I20260926 21:38:04.971053 10232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed: No bootstrap required, opened a new log
I20260926 21:38:04.978452 10232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3215cae2aa6b482ea8673bc617b6b6ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } }
I20260926 21:38:04.978808 10232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:04.978873 10232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3215cae2aa6b482ea8673bc617b6b6ed, State: Initialized, Role: FOLLOWER
I20260926 21:38:04.979437 10232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3215cae2aa6b482ea8673bc617b6b6ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } }
I20260926 21:38:04.979630 10232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:38:04.979730 10232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:38:04.979861 10232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:04.982026 10232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3215cae2aa6b482ea8673bc617b6b6ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } }
I20260926 21:38:04.982559 10232 leader_election.cc:302] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3215cae2aa6b482ea8673bc617b6b6ed; no voters:
I20260926 21:38:04.983345 10232 leader_election.cc:288] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:38:04.983466 10237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:38:04.984184 10237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [term 1 LEADER]: Becoming Leader. State: Replica: 3215cae2aa6b482ea8673bc617b6b6ed, State: Running, Role: LEADER
I20260926 21:38:04.984681 10237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3215cae2aa6b482ea8673bc617b6b6ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } }
I20260926 21:38:04.985489 10232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:38:04.987567 10238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3215cae2aa6b482ea8673bc617b6b6ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3215cae2aa6b482ea8673bc617b6b6ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } } }
I20260926 21:38:04.988093 10238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [sys.catalog]: This master's current role is: LEADER
I20260926 21:38:04.987648 10239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3215cae2aa6b482ea8673bc617b6b6ed. Latest consensus state: current_term: 1 leader_uuid: "3215cae2aa6b482ea8673bc617b6b6ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3215cae2aa6b482ea8673bc617b6b6ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } } }
I20260926 21:38:04.988344 10239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed [sys.catalog]: This master's current role is: LEADER
I20260926 21:38:04.989454 10245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:38:04.994925 10245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:38:05.004073 10245 catalog_manager.cc:1347] Generated new cluster ID: 7ab21dcaf4b84dfebe5c02b041cf2d64
I20260926 21:38:05.004199 10245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:38:05.025240 10245 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:38:05.025638 10245 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:38:05.034564 10245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3215cae2aa6b482ea8673bc617b6b6ed: Generated new TSK 0
I20260926 21:38:05.035370 10245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:38:05.050910 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--builtin_ntp_servers=127.0.97.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 21:38:05.382565 10256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:05.382923 10256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:05.383040 10256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:05.390695 10256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:05.390808 10256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:05.390918 10256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:38:05.402491 10256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:05.404202 10256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:05.406303 10256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:05.419636 10262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:05.419719 10261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:05.419678 10264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:05.420749 10256 server_base.cc:1034] running on GCE node
I20260926 21:38:05.421463 10256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:05.422429 10256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:05.423653 10256 hybrid_clock.cc:648] HybridClock initialized: now 1790458685423614 us; error 44 us; skew 500 ppm
I20260926 21:38:05.423977 10256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:38:05.426263 10256 webserver.cc:460] Webserver started at http://127.0.97.1:38297/ using document root <none> and password file <none>
I20260926 21:38:05.426986 10256 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:05.427120 10256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:05.427454 10256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:05.432072 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "c6cf618e4eab46dc8ea3cd5394d5feac"
format_stamp: "Formatted at 2026-09-26 21:38:05 on dist-test-slave-lhhq"
server_key: "569773d6086149e1c5c6f2e446245918"
server_key_iv: "5ee9293c6e4c1e5240f8db4256cbd9b1"
server_key_version: "encryptionkey@0"
I20260926 21:38:05.432960 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c6cf618e4eab46dc8ea3cd5394d5feac"
format_stamp: "Formatted at 2026-09-26 21:38:05 on dist-test-slave-lhhq"
server_key: "569773d6086149e1c5c6f2e446245918"
server_key_iv: "5ee9293c6e4c1e5240f8db4256cbd9b1"
server_key_version: "encryptionkey@0"
I20260926 21:38:05.438598 10256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:38:05.442168 10270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:05.443593 10256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 21:38:05.443790 10256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "c6cf618e4eab46dc8ea3cd5394d5feac"
format_stamp: "Formatted at 2026-09-26 21:38:05 on dist-test-slave-lhhq"
server_key: "569773d6086149e1c5c6f2e446245918"
server_key_iv: "5ee9293c6e4c1e5240f8db4256cbd9b1"
server_key_version: "encryptionkey@0"
I20260926 21:38:05.444038 10256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:05.449919 10256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:05.450723 10256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:05.451004 10256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:05.451920 10256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:05.453464 10256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:05.453581 10256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:05.453718 10256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:05.453805 10256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:05.487787 10256 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43205
I20260926 21:38:05.487871 10382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43205 every 8 connection(s)
I20260926 21:38:05.489347 10256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:38:05.495090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 10256
I20260926 21:38:05.495289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:38:05.496100 388 external_mini_cluster.cc:1428] Setting key 7cbd59fc224b63cbefecd8ce6c0e7332
I20260926 21:38:05.498271 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:05.491978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53649 (local address 127.0.97.62:46853)
0926 21:38:05.492431 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:05.492436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:05.493653 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 21:38:05.495607 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:05.495613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:05.495684 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:05.495837 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:05.497487 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:05.497493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:05.497497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:05.497518 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:05.497568 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:05.497580 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:05.497711 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:05.497928 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 21:38:05.497993 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":261,"threads_started":1}
I20260926 21:38:05.498944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--builtin_ntp_servers=127.0.97.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 21:38:05.499344 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:05.492521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46853 (local address 127.0.97.1:53649)
0926 21:38:05.493490 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:05.493531 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:05.495240 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:05.496119 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:05.496128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:05.496277 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:05.497116 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:38:05.497119 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:05.497172 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:38:05.497322 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:05.497676 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:05.497687 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:05.497697 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:38:05.498012 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 21:38:05.498294 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":282,"threads_started":1}
I20260926 21:38:05.501097 10383 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46853
I20260926 21:38:05.501437 10383 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:05.501978 10383 heartbeater.cc:511] Master 127.0.97.62:46853 requested a full tablet report, sending...
I20260926 21:38:05.504309 10197 ts_manager.cc:194] Registered new tserver with Master: c6cf618e4eab46dc8ea3cd5394d5feac (127.0.97.1:43205)
W20260926 21:38:05.829798 10387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:05.830152 10387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:05.830313 10387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:05.837574 10387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:05.837718 10387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:05.837918 10387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:38:05.846879 10387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:05.848205 10387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:05.849931 10387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:05.862676 10392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:05.862818 10393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:05.863243 10395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:05.863762 10387 server_base.cc:1034] running on GCE node
I20260926 21:38:05.864425 10387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:05.865415 10387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:05.866660 10387 hybrid_clock.cc:648] HybridClock initialized: now 1790458685866618 us; error 54 us; skew 500 ppm
I20260926 21:38:05.866981 10387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:38:05.869138 10387 webserver.cc:460] Webserver started at http://127.0.97.2:33501/ using document root <none> and password file <none>
I20260926 21:38:05.869854 10387 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:05.869982 10387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:05.870287 10387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:05.873035 10387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "3d93ccbca97b4770b138da3fdaab58da"
format_stamp: "Formatted at 2026-09-26 21:38:05 on dist-test-slave-lhhq"
server_key: "f7499c11cf06992379b9f40783275a49"
server_key_iv: "41eefb2228d5b019cebaa3b8a339b72c"
server_key_version: "encryptionkey@0"
I20260926 21:38:05.873874 10387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3d93ccbca97b4770b138da3fdaab58da"
format_stamp: "Formatted at 2026-09-26 21:38:05 on dist-test-slave-lhhq"
server_key: "f7499c11cf06992379b9f40783275a49"
server_key_iv: "41eefb2228d5b019cebaa3b8a339b72c"
server_key_version: "encryptionkey@0"
I20260926 21:38:05.878840 10387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:38:05.882241 10401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:05.883494 10387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:38:05.883672 10387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "3d93ccbca97b4770b138da3fdaab58da"
format_stamp: "Formatted at 2026-09-26 21:38:05 on dist-test-slave-lhhq"
server_key: "f7499c11cf06992379b9f40783275a49"
server_key_iv: "41eefb2228d5b019cebaa3b8a339b72c"
server_key_version: "encryptionkey@0"
I20260926 21:38:05.883899 10387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:05.889780 10387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:05.890650 10387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:05.890914 10387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:05.891834 10387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:05.893157 10387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:05.893319 10387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:05.893446 10387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:05.893545 10387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:05.926194 10387 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44243
I20260926 21:38:05.926235 10513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44243 every 8 connection(s)
I20260926 21:38:05.927665 10387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:38:05.932545 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 10387
I20260926 21:38:05.932705 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:38:05.933256 388 external_mini_cluster.cc:1428] Setting key dd63b63be52cb3095393de2da90d7063
I20260926 21:38:05.936151 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--builtin_ntp_servers=127.0.97.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 21:38:05.936555 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:05.930394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47261 (local address 127.0.97.62:46853)
0926 21:38:05.930574 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:05.930579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:05.931977 (+ 1398us) server_negotiation.cc:407] Connection header received
0926 21:38:05.934255 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:05.934261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:05.934325 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:05.934441 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:05.935769 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:05.935773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:05.935777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:05.935791 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:05.935826 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:05.935838 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:05.935926 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:05.936209 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 21:38:05.936272 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:38:05.937808 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:05.930937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46853 (local address 127.0.97.2:47261)
0926 21:38:05.931828 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:05.931868 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:05.933999 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:05.934660 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:05.934674 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:05.934767 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:05.935422 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:38:05.935426 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:05.935479 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:38:05.935586 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:05.935939 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:05.935950 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:05.935960 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:38:05.936247 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 21:38:05.936518 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":276,"threads_started":1}
I20260926 21:38:05.939158 10514 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46853
I20260926 21:38:05.939456 10514 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:05.940052 10514 heartbeater.cc:511] Master 127.0.97.62:46853 requested a full tablet report, sending...
I20260926 21:38:05.941665 10197 ts_manager.cc:194] Registered new tserver with Master: 3d93ccbca97b4770b138da3fdaab58da (127.0.97.2:44243)
W20260926 21:38:06.271963 10517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:06.272269 10517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:06.272459 10517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:06.279850 10517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:06.279958 10517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:06.280156 10517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:38:06.289710 10517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:06.291124 10517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:06.292901 10517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:06.305821 10523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:06.305987 10522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:06.305876 10525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:06.306923 10517 server_base.cc:1034] running on GCE node
I20260926 21:38:06.307507 10517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:06.308439 10517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:06.309656 10517 hybrid_clock.cc:648] HybridClock initialized: now 1790458686309611 us; error 55 us; skew 500 ppm
I20260926 21:38:06.309971 10517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:38:06.312252 10517 webserver.cc:460] Webserver started at http://127.0.97.3:41863/ using document root <none> and password file <none>
I20260926 21:38:06.312938 10517 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:06.313068 10517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:06.313442 10517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:06.316273 10517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "e37d5998397d4c85a866df28f5520063"
format_stamp: "Formatted at 2026-09-26 21:38:06 on dist-test-slave-lhhq"
server_key: "d647eed0ef07203faea1467693e75a2c"
server_key_iv: "637efac974011c3b185093145d066bac"
server_key_version: "encryptionkey@0"
I20260926 21:38:06.317073 10517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e37d5998397d4c85a866df28f5520063"
format_stamp: "Formatted at 2026-09-26 21:38:06 on dist-test-slave-lhhq"
server_key: "d647eed0ef07203faea1467693e75a2c"
server_key_iv: "637efac974011c3b185093145d066bac"
server_key_version: "encryptionkey@0"
I20260926 21:38:06.322188 10517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:38:06.325742 10531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:06.327270 10517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 21:38:06.327445 10517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "e37d5998397d4c85a866df28f5520063"
format_stamp: "Formatted at 2026-09-26 21:38:06 on dist-test-slave-lhhq"
server_key: "d647eed0ef07203faea1467693e75a2c"
server_key_iv: "637efac974011c3b185093145d066bac"
server_key_version: "encryptionkey@0"
I20260926 21:38:06.327680 10517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:06.333515 10517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:06.334329 10517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:06.334601 10517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:06.335497 10517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:06.336874 10517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:06.336993 10517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:06.337129 10517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:06.337240 10517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:06.370765 10517 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41197
I20260926 21:38:06.370826 10643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41197 every 8 connection(s)
I20260926 21:38:06.372146 10517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:38:06.380918 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:06.375091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36601 (local address 127.0.97.62:46853)
0926 21:38:06.375305 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:06.375311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:06.376813 (+ 1502us) server_negotiation.cc:407] Connection header received
0926 21:38:06.378580 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:06.378584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:06.378641 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:06.378737 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:06.380187 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:06.380192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:06.380196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:06.380216 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:06.380266 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:06.380281 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:06.380401 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:06.380609 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 21:38:06.380689 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 21:38:06.381901 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:06.375693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46853 (local address 127.0.97.3:36601)
0926 21:38:06.376634 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:06.376682 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:06.378298 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:06.378962 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:06.378971 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:06.379074 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:06.379829 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:38:06.379832 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:06.379873 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:38:06.379966 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:06.380382 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:06.380389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:06.380395 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:38:06.380621 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 21:38:06.380847 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":294,"threads_started":1}
I20260926 21:38:06.382530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 10517
I20260926 21:38:06.382723 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:38:06.383242 388 external_mini_cluster.cc:1428] Setting key fc6dc4fac52d0a15848b6c5cb9cd7006
I20260926 21:38:06.383586 10644 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46853
I20260926 21:38:06.383924 10644 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:06.384609 10644 heartbeater.cc:511] Master 127.0.97.62:46853 requested a full tablet report, sending...
I20260926 21:38:06.386392 10196 ts_manager.cc:194] Registered new tserver with Master: e37d5998397d4c85a866df28f5520063 (127.0.97.3:41197)
I20260926 21:38:06.399008 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 21:38:06.405802 10385 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:38:06.403840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.0.97.62:46853)
0926 21:38:06.404055 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:06.404062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:06.404600 (+ 538us) server_negotiation.cc:407] Connection header received
0926 21:38:06.404845 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:06.404852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:06.404939 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:06.405057 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:06.405516 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35330: BlockingRecv error: recv got EOF from 127.0.0.1:35330 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:38:06.406528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 10256
I20260926 21:38:06.410080 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:06.407570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.0.97.1:43205)
0926 21:38:06.408135 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:06.408144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:06.408182 (+ 38us) server_negotiation.cc:407] Connection header received
0926 21:38:06.408253 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:06.408259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:06.408479 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:06.408606 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:06.409497 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:06.409503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:06.409516 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:06.409527 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:06.409568 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:06.409585 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:06.409689 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:06.409783 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 21:38:06.409867 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":313,"threads_started":1}
I20260926 21:38:06.411286 10378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:06.492619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 10256
I20260926 21:38:06.509439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 10387
I20260926 21:38:06.512776 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:06.510314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.0.97.2:44243)
0926 21:38:06.510830 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:06.510836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:06.510862 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:38:06.510926 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:06.510932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:06.511147 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:06.511279 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:06.512126 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:06.512133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:06.512145 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:06.512157 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:06.512202 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:06.512220 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:06.512323 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:06.512450 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 21:38:06.512533 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":245,"threads_started":1}
I20260926 21:38:06.514045 10509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:06.594419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 10387
I20260926 21:38:06.611176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 10517
I20260926 21:38:06.615135 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:06.612251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.97.3:41197)
0926 21:38:06.613044 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:06.613052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:06.613093 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:38:06.613163 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:06.613169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:06.613489 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:06.613634 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:06.614519 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:06.614525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:06.614536 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:38:06.614552 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:06.614589 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:38:06.614613 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:06.614742 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:06.614793 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 21:38:06.614867 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":460,"threads_started":1}
I20260926 21:38:06.616488 10639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:06.697588 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 10517
I20260926 21:38:06.713531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 10166
I20260926 21:38:06.714779 10227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:06.815832 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 10166
2026-09-26T21:38:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2337 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:38:06 dist-test-slave-lhhq krb5kdc[10662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:38:06 dist-test-slave-lhhq krb5kdc[10662](info): set up 2 sockets
Sep 26 21:38:06 dist-test-slave-lhhq krb5kdc[10662](info): commencing operation
krb5kdc: starting...
W20260926 21:38:07.934139 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:38:08 dist-test-slave-lhhq krb5kdc[10662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:38:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:38:08.168834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37007
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:38:08.507509 10675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:08.507815 10675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:08.507906 10675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:08.515190 10675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:38:08.515322 10675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:08.515370 10675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:08.515429 10675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:38:08.515466 10675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:38:08.525674 10675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:08.526999 10675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:08.528275 10675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:08.539875 10683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:08.539919 10681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:08.539803 10680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:08.541460 10675 server_base.cc:1034] running on GCE node
I20260926 21:38:08.542244 10675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:08.543737 10675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:08.544986 10675 hybrid_clock.cc:648] HybridClock initialized: now 1790458688544931 us; error 49 us; skew 500 ppm
I20260926 21:38:08.545433 10675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:38:08 dist-test-slave-lhhq krb5kdc[10662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:38:08.556167 10675 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:38:08.557914 10675 webserver.cc:460] Webserver started at http://127.0.97.62:38993/ using document root <none> and password file <none>
I20260926 21:38:08.558674 10675 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:08.558777 10675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:08.559037 10675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:08.562156 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "cb63d95197e54ea0962f10e982378fb8"
format_stamp: "Formatted at 2026-09-26 21:38:08 on dist-test-slave-lhhq"
server_key: "3285846e34e1ea48c84a45eecf56b7a4"
server_key_iv: "a00141a079755dec429be4a957eb95f1"
server_key_version: "encryptionkey@0"
I20260926 21:38:08.562922 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "cb63d95197e54ea0962f10e982378fb8"
format_stamp: "Formatted at 2026-09-26 21:38:08 on dist-test-slave-lhhq"
server_key: "3285846e34e1ea48c84a45eecf56b7a4"
server_key_iv: "a00141a079755dec429be4a957eb95f1"
server_key_version: "encryptionkey@0"
I20260926 21:38:08.568333 10675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 21:38:08.571913 10690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:08.573388 10675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:38:08.573577 10675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "cb63d95197e54ea0962f10e982378fb8"
format_stamp: "Formatted at 2026-09-26 21:38:08 on dist-test-slave-lhhq"
server_key: "3285846e34e1ea48c84a45eecf56b7a4"
server_key_iv: "a00141a079755dec429be4a957eb95f1"
server_key_version: "encryptionkey@0"
I20260926 21:38:08.573772 10675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:08.597559 10675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:08.601940 10675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:08.602234 10675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:08.621417 10675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37007
I20260926 21:38:08.621404 10741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37007 every 8 connection(s)
I20260926 21:38:08.622879 10675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:38:08.626829 10742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:08.631408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 10675
I20260926 21:38:08.631620 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:38:08.632412 388 external_mini_cluster.cc:1428] Setting key 18afae441ecbc062e2606fc4e57c9d8e
I20260926 21:38:08.638577 10742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8: Bootstrap starting.
I20260926 21:38:08.642279 10742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:08.643419 10742 log.cc:824] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8: Log is configured to *not* fsync() on all Append() calls
I20260926 21:38:08.646185 10742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8: No bootstrap required, opened a new log
Sep 26 21:38:08 dist-test-slave-lhhq krb5kdc[10662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:38:08.656359 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:08.634578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.0.97.62:37007)
0926 21:38:08.635410 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:08.635423 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:08.635468 (+ 45us) server_negotiation.cc:407] Connection header received
0926 21:38:08.637156 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:08.637179 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:08.637679 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:08.638014 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:08.639740 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:08.641065 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:08.642877 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:08.643600 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:08.643727 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:08.647674 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:08.647684 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:08.647704 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:08.647764 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:08.651815 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:08.652478 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:08.652488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:08.652502 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:08.652709 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:08.653057 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:08.653062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:08.653066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:08.653818 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:08.654120 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:08.654397 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 21:38:08.654880 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":413,"threads_started":1}
I20260926 21:38:08.657128 10742 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb63d95197e54ea0962f10e982378fb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37007 } }
I20260926 21:38:08.657567 10742 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:08.657675 10742 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb63d95197e54ea0962f10e982378fb8, State: Initialized, Role: FOLLOWER
I20260926 21:38:08.658311 10742 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb63d95197e54ea0962f10e982378fb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37007 } }
I20260926 21:38:08.658536 10742 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:38:08.658679 10742 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:38:08.658869 10742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:08.661235 10742 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb63d95197e54ea0962f10e982378fb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37007 } }
I20260926 21:38:08.661734 10742 leader_election.cc:302] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb63d95197e54ea0962f10e982378fb8; no voters:
I20260926 21:38:08.662317 10742 leader_election.cc:288] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:38:08.662436 10747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:38:08.663120 10747 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [term 1 LEADER]: Becoming Leader. State: Replica: cb63d95197e54ea0962f10e982378fb8, State: Running, Role: LEADER
I20260926 21:38:08.663631 10747 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb63d95197e54ea0962f10e982378fb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37007 } }
I20260926 21:38:08.664373 10742 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:38:08.666531 10749 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb63d95197e54ea0962f10e982378fb8. Latest consensus state: current_term: 1 leader_uuid: "cb63d95197e54ea0962f10e982378fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb63d95197e54ea0962f10e982378fb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37007 } } }
I20260926 21:38:08.666603 10748 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb63d95197e54ea0962f10e982378fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb63d95197e54ea0962f10e982378fb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37007 } } }
I20260926 21:38:08.666957 10749 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [sys.catalog]: This master's current role is: LEADER
I20260926 21:38:08.666953 10748 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8 [sys.catalog]: This master's current role is: LEADER
I20260926 21:38:08.668866 10753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:38:08.674180 10753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:38:08.683723 10753 catalog_manager.cc:1347] Generated new cluster ID: b053f798412b4e7cbb5bcd8591524087
I20260926 21:38:08.683858 10753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:38:08.705327 10753 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:38:08.707521 10753 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:38:08.719396 10753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8: Generated new TSK 0
I20260926 21:38:08.720563 10753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:38:08.782187 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37007
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:38:09.116580 10768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:09.116889 10768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:09.117084 10768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:09.124562 10768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:09.124679 10768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:09.124891 10768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:38:09.134824 10768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:09.136214 10768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:09.137583 10768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:09.150686 10774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:09.150722 10773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:09.150763 10776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:09.151541 10768 server_base.cc:1034] running on GCE node
I20260926 21:38:09.152141 10768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:09.153133 10768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:09.154371 10768 hybrid_clock.cc:648] HybridClock initialized: now 1790458689154315 us; error 52 us; skew 500 ppm
I20260926 21:38:09.154745 10768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:38:09 dist-test-slave-lhhq krb5kdc[10662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:38:09.158179 10768 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:38:09.159631 10768 webserver.cc:460] Webserver started at http://127.0.97.1:44311/ using document root <none> and password file <none>
I20260926 21:38:09.160341 10768 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:09.160468 10768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:09.160784 10768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:09.163744 10768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "d97e0c9bbcff4bafad1032262ff81307"
format_stamp: "Formatted at 2026-09-26 21:38:09 on dist-test-slave-lhhq"
server_key: "e67b8d7b6d5fceb6ca77d9f02ec22443"
server_key_iv: "49ce161aba7063cd19a90bbc5b3d2e68"
server_key_version: "encryptionkey@0"
I20260926 21:38:09.164508 10768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d97e0c9bbcff4bafad1032262ff81307"
format_stamp: "Formatted at 2026-09-26 21:38:09 on dist-test-slave-lhhq"
server_key: "e67b8d7b6d5fceb6ca77d9f02ec22443"
server_key_iv: "49ce161aba7063cd19a90bbc5b3d2e68"
server_key_version: "encryptionkey@0"
I20260926 21:38:09.169960 10768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 21:38:09.173465 10783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:09.174863 10768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:38:09.175046 10768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "d97e0c9bbcff4bafad1032262ff81307"
format_stamp: "Formatted at 2026-09-26 21:38:09 on dist-test-slave-lhhq"
server_key: "e67b8d7b6d5fceb6ca77d9f02ec22443"
server_key_iv: "49ce161aba7063cd19a90bbc5b3d2e68"
server_key_version: "encryptionkey@0"
I20260926 21:38:09.175247 10768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:09.191617 10768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:09.195873 10768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:09.196178 10768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:09.197170 10768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:09.198583 10768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:09.198693 10768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:09.198853 10768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:09.198927 10768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:09.232154 10768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41037
I20260926 21:38:09.232205 10895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41037 every 8 connection(s)
I20260926 21:38:09.233580 10768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:38:09.235333 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 10768
I20260926 21:38:09.235538 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:38:09.236399 388 external_mini_cluster.cc:1428] Setting key cc51a7514775e49ce05df3da04e80e69
Sep 26 21:38:09 dist-test-slave-lhhq krb5kdc[10662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:38:09.254204 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:09.236495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57119 (local address 127.0.97.62:37007)
0926 21:38:09.236970 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:09.236975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:09.238160 (+ 1185us) server_negotiation.cc:407] Connection header received
0926 21:38:09.240088 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:09.240093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:09.240185 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:09.240313 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:09.241947 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:09.242650 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:09.244237 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:09.244664 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:09.244813 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:09.248348 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:09.248353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:09.248357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:09.248388 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:09.251050 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:09.251808 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:09.251814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:09.251819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:09.251943 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:09.252385 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:09.252390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:09.252393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:09.252656 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:09.252811 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:09.253611 (+ 800us) server_negotiation.cc:299] Negotiation successful
0926 21:38:09.253833 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":244,"threads_started":1}
I20260926 21:38:09.255223 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:09.237173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37007 (local address 127.0.97.1:57119)
0926 21:38:09.237998 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:09.238036 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:09.239801 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:09.240574 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:09.240584 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:09.241108 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:09.241774 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:09.241791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:09.242808 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:09.242813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:09.244067 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:09.244078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:09.244896 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:09.244905 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:09.245143 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:09.245919 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:09.245943 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:09.248119 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:09.251225 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:09.251239 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:09.251252 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:09.251675 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:09.252056 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:09.252061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:09.252065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:09.252263 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:09.252786 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:09.252804 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:09.253154 (+ 350us) client_negotiation.cc:769] Sending connection context
0926 21:38:09.253515 (+ 361us) client_negotiation.cc:240] Negotiation successful
0926 21:38:09.254076 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":333,"threads_started":1}
I20260926 21:38:09.256880 10896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37007
I20260926 21:38:09.257243 10896 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:38:09.258484 10896 heartbeater.cc:511] Master 127.0.97.62:37007 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:38:09.260860 10707 ts_manager.cc:194] Registered new tserver with Master: d97e0c9bbcff4bafad1032262ff81307 (127.0.97.1:41037)
I20260926 21:38:09.262388 10707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:38:09.288357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37007
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:38:09.624675 10902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:09.625133 10902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:09.625387 10902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:09.633330 10902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:09.633433 10902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:09.633635 10902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:38:09.643410 10902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:09.644683 10902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:09.646061 10902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:09.658353 10907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:09.658438 10908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:09.658610 10910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:09.659765 10902 server_base.cc:1034] running on GCE node
I20260926 21:38:09.660331 10902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:09.661242 10902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:09.662477 10902 hybrid_clock.cc:648] HybridClock initialized: now 1790458689662420 us; error 41 us; skew 500 ppm
I20260926 21:38:09.662829 10902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:38:09 dist-test-slave-lhhq krb5kdc[10662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:38:09.666002 10902 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:38:09.667372 10902 webserver.cc:460] Webserver started at http://127.0.97.2:39341/ using document root <none> and password file <none>
I20260926 21:38:09.668051 10902 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:09.668179 10902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:09.668478 10902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:09.671430 10902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "bf4eee8429ba440cb8b8177c3222ecb5"
format_stamp: "Formatted at 2026-09-26 21:38:09 on dist-test-slave-lhhq"
server_key: "5b49d8623f65b1c851a32b78c5cf7ea6"
server_key_iv: "3891e911d332158cfb278f6e9d9fceb3"
server_key_version: "encryptionkey@0"
I20260926 21:38:09.672184 10902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bf4eee8429ba440cb8b8177c3222ecb5"
format_stamp: "Formatted at 2026-09-26 21:38:09 on dist-test-slave-lhhq"
server_key: "5b49d8623f65b1c851a32b78c5cf7ea6"
server_key_iv: "3891e911d332158cfb278f6e9d9fceb3"
server_key_version: "encryptionkey@0"
I20260926 21:38:09.677266 10902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:38:09.680784 10917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:09.682051 10902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 21:38:09.682216 10902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "bf4eee8429ba440cb8b8177c3222ecb5"
format_stamp: "Formatted at 2026-09-26 21:38:09 on dist-test-slave-lhhq"
server_key: "5b49d8623f65b1c851a32b78c5cf7ea6"
server_key_iv: "3891e911d332158cfb278f6e9d9fceb3"
server_key_version: "encryptionkey@0"
I20260926 21:38:09.682403 10902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:09.708406 10902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:09.712831 10902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:09.713160 10902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:09.714191 10902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:09.715636 10902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:09.715749 10902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:09.715912 10902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:09.715991 10902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:09.753314 10902 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42381
I20260926 21:38:09.753367 11029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42381 every 8 connection(s)
I20260926 21:38:09.754611 10902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:38:09.762065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 10902
I20260926 21:38:09.762347 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:38:09.762797 388 external_mini_cluster.cc:1428] Setting key 7163f248154f9be27b890152efe5548c
Sep 26 21:38:09 dist-test-slave-lhhq krb5kdc[10662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:38:09.775671 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:09.757083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55539 (local address 127.0.97.62:37007)
0926 21:38:09.757550 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:09.757554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:09.758434 (+ 880us) server_negotiation.cc:407] Connection header received
0926 21:38:09.760376 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:09.760385 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:09.760475 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:09.760592 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:09.762148 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:09.763005 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:09.764872 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:09.765411 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:09.765539 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:09.769288 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:09.769294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:09.769297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:09.769328 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:09.772297 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:09.773170 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:09.773174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:09.773223 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:09.773357 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:09.773863 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:09.773867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:09.773869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:09.774127 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:09.774246 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:09.775206 (+ 960us) server_negotiation.cc:299] Negotiation successful
0926 21:38:09.775396 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":257,"threads_started":1}
I20260926 21:38:09.777259 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:09.757581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37007 (local address 127.0.97.2:55539)
0926 21:38:09.758267 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:09.758305 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:09.760072 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:09.760829 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:09.760837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:09.761405 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:09.761969 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:09.761986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:09.763156 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:09.763160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:09.764720 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:09.764736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:09.765644 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:09.765653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:09.765875 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:09.766575 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:09.766597 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:09.769019 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:09.772462 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:09.772470 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:09.772484 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:09.773032 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:09.773466 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:09.773473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:09.773479 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:09.773746 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:09.774267 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:09.774279 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:09.774765 (+ 486us) client_negotiation.cc:769] Sending connection context
0926 21:38:09.775089 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 21:38:09.775692 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":267,"threads_started":1}
I20260926 21:38:09.778946 11030 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37007
I20260926 21:38:09.779284 11030 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:09.780203 11030 heartbeater.cc:511] Master 127.0.97.62:37007 requested a full tablet report, sending...
I20260926 21:38:09.781949 10707 ts_manager.cc:194] Registered new tserver with Master: bf4eee8429ba440cb8b8177c3222ecb5 (127.0.97.2:42381)
I20260926 21:38:09.783274 10707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55539
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:38:09.816612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37007
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:38:10.134008 11036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:10.134272 11036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:10.134363 11036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:10.141302 11036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:10.141398 11036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:10.141505 11036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:38:10.151008 11036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:10.152199 11036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:10.153494 11036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:10.166160 11042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:10.166203 11044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:10.166450 11041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:10.167292 11036 server_base.cc:1034] running on GCE node
I20260926 21:38:10.167889 11036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:10.168743 11036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:10.169929 11036 hybrid_clock.cc:648] HybridClock initialized: now 1790458690169884 us; error 34 us; skew 500 ppm
I20260926 21:38:10.170253 11036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:38:10 dist-test-slave-lhhq krb5kdc[10662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:38:10.173456 11036 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:38:10.174870 11036 webserver.cc:460] Webserver started at http://127.0.97.3:41305/ using document root <none> and password file <none>
I20260926 21:38:10.175523 11036 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:10.175668 11036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:10.175979 11036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:38:10.179090 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "743880623a8347c5b160c98964874fb1"
format_stamp: "Formatted at 2026-09-26 21:38:10 on dist-test-slave-lhhq"
server_key: "da7928d9e03ca94669df8c8d2b106fe5"
server_key_iv: "e0186ec60c050b7c44bb2402cd38d104"
server_key_version: "encryptionkey@0"
I20260926 21:38:10.179837 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "743880623a8347c5b160c98964874fb1"
format_stamp: "Formatted at 2026-09-26 21:38:10 on dist-test-slave-lhhq"
server_key: "da7928d9e03ca94669df8c8d2b106fe5"
server_key_iv: "e0186ec60c050b7c44bb2402cd38d104"
server_key_version: "encryptionkey@0"
I20260926 21:38:10.184855 11036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:38:10.188256 11051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:10.189642 11036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 21:38:10.189827 11036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "743880623a8347c5b160c98964874fb1"
format_stamp: "Formatted at 2026-09-26 21:38:10 on dist-test-slave-lhhq"
server_key: "da7928d9e03ca94669df8c8d2b106fe5"
server_key_iv: "e0186ec60c050b7c44bb2402cd38d104"
server_key_version: "encryptionkey@0"
I20260926 21:38:10.190016 11036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:10.207396 11036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:10.211582 11036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:10.211879 11036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:10.212803 11036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:10.214159 11036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:10.214263 11036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:10.214416 11036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:10.214488 11036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:10.246971 11036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37217
I20260926 21:38:10.247052 11163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37217 every 8 connection(s)
I20260926 21:38:10.248196 11036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:38:10.249117 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 11036
I20260926 21:38:10.249305 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:38:10.249866 388 external_mini_cluster.cc:1428] Setting key f05302f3ca16836c43f5a6a7013a45cf
Sep 26 21:38:10 dist-test-slave-lhhq krb5kdc[10662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:38:10.265470 10896 heartbeater.cc:503] Master 127.0.97.62:37007 was elected leader, sending a full tablet report...
I20260926 21:38:10.268887 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.250847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36837 (local address 127.0.97.62:37007)
0926 21:38:10.251041 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.251046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.252495 (+ 1449us) server_negotiation.cc:407] Connection header received
0926 21:38:10.254484 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.254489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.254586 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.254708 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:10.256132 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.256854 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.258424 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.258820 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.258949 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.262312 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:10.262317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:10.262320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:10.262350 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:10.265674 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.266613 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.266616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.266619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.266738 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.267269 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.267273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.267276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.267569 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:10.267724 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.268434 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.268637 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:38:10.269979 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.251583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37007 (local address 127.0.97.3:36837)
0926 21:38:10.252351 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:10.252388 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:10.254236 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:10.254943 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:10.254952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:10.255433 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:10.255976 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.255992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.256998 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.257004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.258280 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.258291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.259028 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.259034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.259262 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.259934 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:10.259955 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:10.262128 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:10.265860 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:10.265869 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:10.265891 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:10.266488 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:10.266856 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:10.266861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:10.266865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:10.267094 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:10.267719 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:10.267724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:10.268061 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 21:38:10.268292 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 21:38:10.268724 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":300,"threads_started":1}
I20260926 21:38:10.271266 11164 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37007
I20260926 21:38:10.271517 11164 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:10.272189 11164 heartbeater.cc:511] Master 127.0.97.62:37007 requested a full tablet report, sending...
I20260926 21:38:10.273450 10707 ts_manager.cc:194] Registered new tserver with Master: 743880623a8347c5b160c98964874fb1 (127.0.97.3:37217)
I20260926 21:38:10.274231 10707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36837
I20260926 21:38:10.278550 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:38:10.295522 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.283394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.0.97.62:37007)
0926 21:38:10.283575 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.283580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.284009 (+ 429us) server_negotiation.cc:407] Connection header received
0926 21:38:10.284142 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.284145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.284202 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.284288 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:10.285802 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.286520 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.287716 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.288082 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.288172 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.290301 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:10.290306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:10.290309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:10.290336 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:10.292838 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.293566 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.293573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.293576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.293707 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.294102 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.294107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.294110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.294414 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:10.294667 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.295001 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.295213 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:38:10.299441 10707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:38:10.307687 10707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:38:10.331482 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.321366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.0.97.1:41037)
0926 21:38:10.322722 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.322729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.322759 (+ 30us) server_negotiation.cc:407] Connection header received
0926 21:38:10.322830 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.322835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.323012 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.323202 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:10.323889 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.325803 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.328535 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.330026 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.330117 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.330349 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.331003 (+ 654us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.331162 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":1102,"threads_started":1}
I20260926 21:38:10.331517 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.320915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:56910)
0926 21:38:10.321916 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:10.321959 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:10.322110 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:10.323209 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:10.323216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:10.323260 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:10.323689 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.323701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.325925 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.325930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.328332 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.328349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.330513 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.330520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.330760 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.330814 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 21:38:10.330953 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 21:38:10.331125 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":4416,"threads_started":1}
I20260926 21:38:10.332757 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.321223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37217 (local address 127.0.0.1:55144)
0926 21:38:10.325533 (+ 4310us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:10.325547 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:10.325673 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:10.326496 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:10.326502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:10.326538 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:10.327023 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.327035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.328779 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.328784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.330094 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.330104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.331672 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.331679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.331870 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.332197 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 21:38:10.332285 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 21:38:10.332402 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4220,"thread_start_us":322,"threads_started":1}
I20260926 21:38:10.333535 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.321468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.0.97.3:37217)
0926 21:38:10.323860 (+ 2392us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.323867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.325636 (+ 1769us) server_negotiation.cc:407] Connection header received
0926 21:38:10.325827 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.325834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.326083 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.326266 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:10.327228 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.328616 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.330246 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.331207 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.331327 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.331533 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.332970 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.333148 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2209,"thread_start_us":2046,"threads_started":1}
I20260926 21:38:10.335820 10831 tablet_service.cc:1467] Processing CreateTablet for tablet a03a63075fb444449045f0cadaf360d0 (DEFAULT_TABLE table=test-table [id=35e183d697994b1da5de6e1863b5a4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:38:10.336436 11099 tablet_service.cc:1467] Processing CreateTablet for tablet a03a63075fb444449045f0cadaf360d0 (DEFAULT_TABLE table=test-table [id=35e183d697994b1da5de6e1863b5a4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:38:10.337145 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03a63075fb444449045f0cadaf360d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:10.337437 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.325936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.0.97.2:42381)
0926 21:38:10.326483 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.326491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.327235 (+ 744us) server_negotiation.cc:407] Connection header received
0926 21:38:10.327518 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.327525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.327717 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.327918 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:10.328777 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.330275 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.332702 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.333969 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.334048 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.334163 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.336932 (+ 2769us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.337099 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":306,"threads_started":1}
I20260926 21:38:10.337852 11099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03a63075fb444449045f0cadaf360d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:10.338258 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.326495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42381 (local address 127.0.0.1:47826)
0926 21:38:10.327105 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:10.327118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:10.327340 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:10.328127 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:10.328132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:10.328161 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:10.328603 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.328615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.330450 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.330454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.332534 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.332547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.336526 (+ 3979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.336534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.336716 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.336757 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 21:38:10.337754 (+ 997us) client_negotiation.cc:240] Negotiation successful
0926 21:38:10.337887 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":1,"thread_start_us":362,"threads_started":1}
I20260926 21:38:10.341956 10965 tablet_service.cc:1467] Processing CreateTablet for tablet a03a63075fb444449045f0cadaf360d0 (DEFAULT_TABLE table=test-table [id=35e183d697994b1da5de6e1863b5a4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:38:10.343400 10965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03a63075fb444449045f0cadaf360d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:10.349171 11182 tablet_bootstrap.cc:492] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: Bootstrap starting.
I20260926 21:38:10.352865 11182 tablet_bootstrap.cc:654] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:10.353925 11182 log.cc:824] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: Log is configured to *not* fsync() on all Append() calls
I20260926 21:38:10.356667 11182 tablet_bootstrap.cc:492] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: No bootstrap required, opened a new log
I20260926 21:38:10.356933 11182 ts_tablet_manager.cc:1397] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260926 21:38:10.357259 11183 tablet_bootstrap.cc:492] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: Bootstrap starting.
I20260926 21:38:10.358073 11185 tablet_bootstrap.cc:492] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: Bootstrap starting.
I20260926 21:38:10.360834 11183 tablet_bootstrap.cc:654] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:10.361519 11185 tablet_bootstrap.cc:654] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:10.361997 11183 log.cc:824] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: Log is configured to *not* fsync() on all Append() calls
I20260926 21:38:10.362660 11185 log.cc:824] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: Log is configured to *not* fsync() on all Append() calls
I20260926 21:38:10.364557 11183 tablet_bootstrap.cc:492] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: No bootstrap required, opened a new log
I20260926 21:38:10.364821 11183 ts_tablet_manager.cc:1397] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 21:38:10.367691 11185 tablet_bootstrap.cc:492] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: No bootstrap required, opened a new log
I20260926 21:38:10.367954 11185 ts_tablet_manager.cc:1397] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 21:38:10.368907 11182 raft_consensus.cc:348] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } }
I20260926 21:38:10.369395 11182 raft_consensus.cc:374] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:10.369499 11182 raft_consensus.cc:729] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743880623a8347c5b160c98964874fb1, State: Initialized, Role: FOLLOWER
I20260926 21:38:10.370124 11182 consensus_queue.cc:260] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } }
I20260926 21:38:10.373082 11183 raft_consensus.cc:348] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } }
I20260926 21:38:10.373476 11183 raft_consensus.cc:374] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:10.373590 11183 raft_consensus.cc:729] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97e0c9bbcff4bafad1032262ff81307, State: Initialized, Role: FOLLOWER
I20260926 21:38:10.374153 11183 consensus_queue.cc:260] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } }
I20260926 21:38:10.376861 11183 ts_tablet_manager.cc:1428] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260926 21:38:10.377169 11164 heartbeater.cc:503] Master 127.0.97.62:37007 was elected leader, sending a full tablet report...
I20260926 21:38:10.378062 11182 ts_tablet_manager.cc:1428] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: Time spent starting tablet: real 0.021s user 0.016s sys 0.005s
I20260926 21:38:10.380272 11185 raft_consensus.cc:348] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } }
I20260926 21:38:10.380628 11185 raft_consensus.cc:374] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:10.380735 11185 raft_consensus.cc:729] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4eee8429ba440cb8b8177c3222ecb5, State: Initialized, Role: FOLLOWER
I20260926 21:38:10.381373 11185 consensus_queue.cc:260] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } }
I20260926 21:38:10.382325 11030 heartbeater.cc:503] Master 127.0.97.62:37007 was elected leader, sending a full tablet report...
I20260926 21:38:10.382725 11185 ts_tablet_manager.cc:1428] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: Time spent starting tablet: real 0.015s user 0.004s sys 0.012s
I20260926 21:38:10.412796 11190 raft_consensus.cc:482] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:38:10.413065 11190 raft_consensus.cc:504] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } }
I20260926 21:38:10.414805 11190 leader_election.cc:288] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d97e0c9bbcff4bafad1032262ff81307 (127.0.97.1:41037), 743880623a8347c5b160c98964874fb1 (127.0.97.3:37217)
I20260926 21:38:10.428615 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.415234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49807 (local address 127.0.97.1:41037)
0926 21:38:10.415529 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.415539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.416309 (+ 770us) server_negotiation.cc:407] Connection header received
0926 21:38:10.416538 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.416543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.416608 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.416705 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:10.422466 (+ 5761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.423520 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.426622 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.427707 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.427817 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.427896 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.428226 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.428354 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260926 21:38:10.428740 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.415202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.97.2:49807)
0926 21:38:10.416187 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:10.416203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:10.416381 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:10.416745 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:10.416752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:10.416775 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:10.422229 (+ 5454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.422250 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.423754 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.423759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.426419 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.426434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.427909 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.427914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.428023 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.428049 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 21:38:10.428169 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 21:38:10.428330 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":99,"spinlock_wait_cycles":28928,"thread_start_us":625,"threads_started":1}
I20260926 21:38:10.429621 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.415327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36269 (local address 127.0.97.3:37217)
0926 21:38:10.415552 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.415561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.416378 (+ 817us) server_negotiation.cc:407] Connection header received
0926 21:38:10.416561 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.416565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.416621 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.416717 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:10.422772 (+ 6055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.423849 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.427990 (+ 4141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.428826 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.428945 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.429053 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.429320 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.429415 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 21:38:10.429845 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.415322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37217 (local address 127.0.97.2:36269)
0926 21:38:10.416288 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:10.416302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:10.416406 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:10.416747 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:10.416752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:10.416844 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:10.422531 (+ 5687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.422551 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.424040 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.424046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.426426 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:10.426434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.429007 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.429011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:10.429131 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.429154 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:38:10.429296 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 21:38:10.429437 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":44,"spinlock_wait_cycles":25472,"thread_start_us":466,"threads_started":1}
I20260926 21:38:10.430182 10851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03a63075fb444449045f0cadaf360d0" candidate_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97e0c9bbcff4bafad1032262ff81307" is_pre_election: true
I20260926 21:38:10.430738 10851 raft_consensus.cc:2454] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf4eee8429ba440cb8b8177c3222ecb5 in term 0.
I20260926 21:38:10.430886 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03a63075fb444449045f0cadaf360d0" candidate_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "743880623a8347c5b160c98964874fb1" is_pre_election: true
I20260926 21:38:10.431414 11119 raft_consensus.cc:2454] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf4eee8429ba440cb8b8177c3222ecb5 in term 0.
I20260926 21:38:10.431450 10919 leader_election.cc:302] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf4eee8429ba440cb8b8177c3222ecb5, d97e0c9bbcff4bafad1032262ff81307; no voters:
I20260926 21:38:10.431911 11190 raft_consensus.cc:2781] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:38:10.432070 11190 raft_consensus.cc:482] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:38:10.432175 11190 raft_consensus.cc:3037] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:10.434424 11190 raft_consensus.cc:504] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } }
I20260926 21:38:10.434981 11190 leader_election.cc:288] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [CANDIDATE]: Term 1 election: Requested vote from peers d97e0c9bbcff4bafad1032262ff81307 (127.0.97.1:41037), 743880623a8347c5b160c98964874fb1 (127.0.97.3:37217)
I20260926 21:38:10.435384 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03a63075fb444449045f0cadaf360d0" candidate_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "743880623a8347c5b160c98964874fb1"
I20260926 21:38:10.435536 11119 raft_consensus.cc:3037] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:10.437472 10851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03a63075fb444449045f0cadaf360d0" candidate_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97e0c9bbcff4bafad1032262ff81307"
I20260926 21:38:10.437688 10851 raft_consensus.cc:3037] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:10.437718 11119 raft_consensus.cc:2454] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf4eee8429ba440cb8b8177c3222ecb5 in term 1.
I20260926 21:38:10.438148 10920 leader_election.cc:302] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 743880623a8347c5b160c98964874fb1, bf4eee8429ba440cb8b8177c3222ecb5; no voters:
I20260926 21:38:10.438453 11190 raft_consensus.cc:2781] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:38:10.438938 11190 raft_consensus.cc:686] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 1 LEADER]: Becoming Leader. State: Replica: bf4eee8429ba440cb8b8177c3222ecb5, State: Running, Role: LEADER
I20260926 21:38:10.439476 11190 consensus_queue.cc:237] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } }
I20260926 21:38:10.440850 10851 raft_consensus.cc:2454] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf4eee8429ba440cb8b8177c3222ecb5 in term 1.
I20260926 21:38:10.443851 10706 catalog_manager.cc:5495] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf4eee8429ba440cb8b8177c3222ecb5 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf4eee8429ba440cb8b8177c3222ecb5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 } health_report { overall_health: UNKNOWN } } }
I20260926 21:38:10.461512 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.456486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.0.97.2:42381)
0926 21:38:10.456692 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.456697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.456715 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:38:10.456944 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.456949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.457025 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.457165 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:38:10.457530 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.458232 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.459452 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.459852 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.459941 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.460369 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:38:10.460917 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:38:10.461019 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.461157 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.461294 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:38:10.469908 11119 raft_consensus.cc:1261] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [term 1 FOLLOWER]: Refusing update from remote peer bf4eee8429ba440cb8b8177c3222ecb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:38:10.469921 10851 raft_consensus.cc:1261] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [term 1 FOLLOWER]: Refusing update from remote peer bf4eee8429ba440cb8b8177c3222ecb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:38:10.470899 11193 consensus_queue.cc:1035] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97e0c9bbcff4bafad1032262ff81307" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:38:10.471267 11190 consensus_queue.cc:1035] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "743880623a8347c5b160c98964874fb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:38:10.495847 10707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:38:10.496153 10707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:38:10.499769 10707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb63d95197e54ea0962f10e982378fb8: Sending DeleteTablet for 3 replicas of tablet a03a63075fb444449045f0cadaf360d0
I20260926 21:38:10.501334 11099 tablet_service.cc:1514] Processing DeleteTablet for tablet a03a63075fb444449045f0cadaf360d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:38:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55144
I20260926 21:38:10.501339 10965 tablet_service.cc:1514] Processing DeleteTablet for tablet a03a63075fb444449045f0cadaf360d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:38:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47826
I20260926 21:38:10.501358 10831 tablet_service.cc:1514] Processing DeleteTablet for tablet a03a63075fb444449045f0cadaf360d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:38:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56910
I20260926 21:38:10.503746 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 10768
I20260926 21:38:10.514806 11207 tablet_replica.cc:330] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: stopping tablet replica
I20260926 21:38:10.515399 11208 tablet_replica.cc:330] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: stopping tablet replica
I20260926 21:38:10.516014 11208 raft_consensus.cc:2229] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:38:10.516547 11207 raft_consensus.cc:2229] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:38:10.517314 11207 raft_consensus.cc:2258] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:38:10.517867 11208 raft_consensus.cc:2258] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:38:10.520407 11208 ts_tablet_manager.cc:1905] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:38:10.522361 11207 ts_tablet_manager.cc:1905] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:38:10.522944 11206 tablet_replica.cc:330] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: stopping tablet replica
I20260926 21:38:10.523353 11206 raft_consensus.cc:2229] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:38:10.523833 11206 raft_consensus.cc:2258] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:38:10.525779 11206 ts_tablet_manager.cc:1905] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:38:10.530597 11207 ts_tablet_manager.cc:1918] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:38:10.530782 11207 log.cc:1196] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/a03a63075fb444449045f0cadaf360d0
I20260926 21:38:10.531378 11207 ts_tablet_manager.cc:1939] T a03a63075fb444449045f0cadaf360d0 P bf4eee8429ba440cb8b8177c3222ecb5: Deleting consensus metadata
I20260926 21:38:10.532531 11206 ts_tablet_manager.cc:1918] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:38:10.532687 11206 log.cc:1196] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/a03a63075fb444449045f0cadaf360d0
I20260926 21:38:10.533353 10692 catalog_manager.cc:4841] TS bf4eee8429ba440cb8b8177c3222ecb5 (127.0.97.2:42381): tablet a03a63075fb444449045f0cadaf360d0 (table test-table [id=35e183d697994b1da5de6e1863b5a4f6]) successfully deleted
Sep 26 21:38:10 dist-test-slave-lhhq krb5kdc[10662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:38:10.533890 11206 ts_tablet_manager.cc:1939] T a03a63075fb444449045f0cadaf360d0 P 743880623a8347c5b160c98964874fb1: Deleting consensus metadata
I20260926 21:38:10.535276 11208 ts_tablet_manager.cc:1918] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:38:10.535456 11208 log.cc:1196] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/a03a63075fb444449045f0cadaf360d0
I20260926 21:38:10.535611 10693 catalog_manager.cc:4841] TS 743880623a8347c5b160c98964874fb1 (127.0.97.3:37217): tablet a03a63075fb444449045f0cadaf360d0 (table test-table [id=35e183d697994b1da5de6e1863b5a4f6]) successfully deleted
I20260926 21:38:10.536000 11208 ts_tablet_manager.cc:1939] T a03a63075fb444449045f0cadaf360d0 P d97e0c9bbcff4bafad1032262ff81307: Deleting consensus metadata
I20260926 21:38:10.537803 10692 catalog_manager.cc:4841] TS d97e0c9bbcff4bafad1032262ff81307 (127.0.97.1:41037): tablet a03a63075fb444449045f0cadaf360d0 (table test-table [id=35e183d697994b1da5de6e1863b5a4f6]) successfully deleted
I20260926 21:38:10.541706 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.505518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.0.97.1:41037)
0926 21:38:10.505743 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.505749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.515084 (+ 9335us) server_negotiation.cc:407] Connection header received
0926 21:38:10.515427 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.515436 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.515560 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.515652 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:10.517973 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.519550 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.522118 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.522840 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.522912 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.535002 (+ 12090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:10.535020 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:10.535039 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:10.535094 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:10.538807 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.539673 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.539680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.539682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.539800 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.540225 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.540228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.540230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.540577 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:10.540786 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.541120 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.541404 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:38:10.542374 10891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:10.657531 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 10768
I20260926 21:38:10.675354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 10902
Sep 26 21:38:10 dist-test-slave-lhhq krb5kdc[10662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:38:10.692615 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.676164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.97.2:42381)
0926 21:38:10.676335 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.676339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.676409 (+ 70us) server_negotiation.cc:407] Connection header received
0926 21:38:10.676589 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.676612 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.676671 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.676757 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:10.678341 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.679131 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.680899 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.681388 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.681487 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.686474 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:10.686498 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:10.686514 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:10.686558 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:10.689946 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.690615 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.690628 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.690630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.690772 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.691130 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.691136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.691139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.691488 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:10.691863 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.692051 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.692273 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:38:10.693418 11025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:10.805568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 10902
I20260926 21:38:10.824312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 11036
Sep 26 21:38:10 dist-test-slave-lhhq krb5kdc[10662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:38:10.839326 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:10.825117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.0.97.3:37217)
0926 21:38:10.825342 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:10.825349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:10.825370 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:38:10.825564 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:10.825569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:10.825646 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:10.825766 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:10.827002 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.827722 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.828953 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:10.829442 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:10.829571 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:10.833761 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:10.833777 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:10.833792 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:10.833827 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:10.836373 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.837007 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.837023 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.837026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.837160 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:10.837808 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:10.837813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:10.837818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:10.838148 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:10.838459 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:10.838777 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 21:38:10.838995 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 21:38:10.839974 11159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:10.928419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 11036
I20260926 21:38:10.945811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 10675
I20260926 21:38:10.947002 10737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:11.056221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 10675
2026-09-26T21:38:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4242 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:38:11 dist-test-slave-lhhq krb5kdc[11216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:38:11 dist-test-slave-lhhq krb5kdc[11216](info): set up 2 sockets
Sep 26 21:38:11 dist-test-slave-lhhq krb5kdc[11216](info): commencing operation
krb5kdc: starting...
W20260926 21:38:12.178040 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:38:12 dist-test-slave-lhhq krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:38:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 21:38:12.511592 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh3AZIa/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/postgres -l logfile start
2026-09-26 21:38:15.139 UTC [11246] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 21:38:15.139 UTC [11246] LOG: listening on IPv4 address "127.0.97.20", port 57309
2026-09-26 21:38:15.143 UTC [11246] LOG: listening on Unix socket "/tmp/.s.PGSQL.57309"
2026-09-26 21:38:15.165 UTC [11249] LOG: database system was shut down at 2026-09-26 21:38:13 UTC
2026-09-26 21:38:15.168 UTC [11246] LOG: database system is ready to accept connections
I20260926 21:38:16.197743 388 mini_postgres.cc:96] Postgres bound to 57309
2026-09-26 21:38:16.215 UTC [11257] FATAL: database "slave" does not exist
127.0.97.20:57309 - accepting connections
I20260926 21:38:16.215857 388 mini_ranger.cc:161] Starting Ranger...
I20260926 21:38:16.243186 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 21:38:16.763093 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 21:38:16.763230 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-admin
2026-09-26 21:38:17,084 [I] DB FLAVOR :POSTGRES
2026-09-26 21:38:17,084 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 21:38:17,085 [I] Checking connection..
2026-09-26 21:38:17,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 21:38:17,355 [I] Checking connection passed.
2026-09-26 21:38:17,355 [I] --------- Verifying version history table ---------
2026-09-26 21:38:17,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:38:17,620 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 21:38:17,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:38:17,878 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 21:38:17,879 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 21:38:17,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 21:38:18.118 UTC [11321] WARNING: there is no transaction in progress
2026-09-26 21:38:18,124 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 21:38:18,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:38:18,381 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 21:38:18,381 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 21:38:18,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 21:38:18,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 21:38:18,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lhhq.c.gcp-upstream.internal','N') ;"
2026-09-26 21:38:19,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 21:38:19,390 [I] Table x_portal_user does not exist in database ranger
2026-09-26 21:38:19,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 21:38:19,647 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 21:38:19,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 21:38:19,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 21:38:20,150 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 21:38:20,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 21:38:21.088 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.104 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.120 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.136 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.152 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.169 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.222 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.229 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.244 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.254 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.264 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.277 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.286 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.293 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.299 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21.306 UTC [11429] WARNING: there is no transaction in progress
2026-09-26 21:38:21,946 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 21:38:21,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 21:38:22,190 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 21:38:22,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 21:38:22,434 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 21:38:22,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 21:38:22,676 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 21:38:22,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 21:38:22,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 21:38:23,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lhhq.c.gcp-upstream.internal';"
2026-09-26 21:38:23,402 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 21:38:23,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 21:38:23,651 [I] DB_PATCHES have already been applied
I20260926 21:38:23.658073 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh3AZIa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 21:38:23.658190 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 21:38:23.674772 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:38985
Sep 26, 2026 9:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:38:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 21:38:24.831887 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.616s user 0.000s sys 0.043s
Sep 26, 2026 9:38:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:38:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 9:38:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:38:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:38:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 21:38:46.858162 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 21:38:47 dist-test-slave-lhhq krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458727, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 21:38:47.026415 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 21:38:47.047048 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 21:38:47.912686 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 21:38:47.912806 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-kms
I20260926 21:38:47.912962 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:57309
2026-09-26 21:38:48,348 [I] DB FLAVOR :POSTGRES
2026-09-26 21:38:48,349 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 21:38:48,349 [I] Checking connection
2026-09-26 21:38:48,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 21:38:48,583 [I] connection success
2026-09-26 21:38:48,583 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 21:38:48,583 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 21:38:48,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 21:38:48,852 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 21:38:48,852 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 21:38:48,852 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 21:38:48,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 21:38:49,125 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 21:38:49.558655 388 mini_ranger_kms.cc:310] Created kms service
I20260926 21:38:49.772742 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 21:38:49.819248 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 21:38:49 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458727, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:38:50.198765 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 21:38:50.199117 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 21:38:50.199226 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 21:38:50.216377 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11609
I20260926 21:38:50.216593 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:58171
Sep 26, 2026 9:38:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:38:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:38:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:38:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:38:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:38:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 21:38:52 dist-test-slave-lhhq krb5kdc[11216](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458732, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:38:52.582016 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.556s user 0.120s sys 0.332s
I20260926 21:38:52.582304 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 21:38:52.582520 388 mini_ranger_kms.cc:401] 127.0.97.20:58171/kms/v1/keys
Sep 26 21:38:52 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458732, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:38:53 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458732, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:38:55 dist-test-slave-lhhq krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:38:55.629344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45107
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45107
--ranger_config_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58171/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:38:55.972352 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:55.972754 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:55.972926 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:55.980141 11660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:38:55.980316 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:55.980424 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:55.980525 11660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:38:55.980614 11660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:38:55.991024 11660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:58171/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45107
--ranger_config_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:55.992863 11660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:55.994261 11660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:56.006466 11668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:56.006547 11665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:56.006515 11666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:56.007663 11660 server_base.cc:1034] running on GCE node
I20260926 21:38:56.008440 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:56.009943 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:56.011193 11660 hybrid_clock.cc:648] HybridClock initialized: now 1790458736011153 us; error 53 us; skew 500 ppm
I20260926 21:38:56.011560 11660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:38:56 dist-test-slave-lhhq krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:38:56.023483 11660 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:38:56.025101 11660 webserver.cc:460] Webserver started at http://127.0.97.62:35537/ using document root <none> and password file <none>
I20260926 21:38:56.025877 11660 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:56.026015 11660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:56.026329 11660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:38:56 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:38:56.080747 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "92d40f3b0d3e49d6a21e1c98f250a892"
format_stamp: "Formatted at 2026-09-26 21:38:56 on dist-test-slave-lhhq"
server_key: "31c12330f3c79012dce8497071a0d702"
server_key_iv: "29ad98e15c07d80c724a21b37257e653"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:56.081692 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "92d40f3b0d3e49d6a21e1c98f250a892"
format_stamp: "Formatted at 2026-09-26 21:38:56 on dist-test-slave-lhhq"
server_key: "31c12330f3c79012dce8497071a0d702"
server_key_iv: "29ad98e15c07d80c724a21b37257e653"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:56.087206 11660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:38:56.103395 11675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:56.104933 11660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:38:56.105108 11660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "92d40f3b0d3e49d6a21e1c98f250a892"
format_stamp: "Formatted at 2026-09-26 21:38:56 on dist-test-slave-lhhq"
server_key: "31c12330f3c79012dce8497071a0d702"
server_key_iv: "29ad98e15c07d80c724a21b37257e653"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:56.105396 11660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:56.131814 11660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:56.136960 11660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:56.137390 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:56.157776 11660 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45107
I20260926 21:38:56.157805 11726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45107 every 8 connection(s)
I20260926 21:38:56.159202 11660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:38:56.163594 11727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:56.166594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 11660
I20260926 21:38:56.166807 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:38:56.167419 388 external_mini_cluster.cc:1428] Setting key 1beb091ad9edba38f6c2635a5b8afd28
I20260926 21:38:56.175112 11727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892: Bootstrap starting.
I20260926 21:38:56.178537 11727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:56.179474 11727 log.cc:824] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:38:56 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:38:56.182446 11727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892: No bootstrap required, opened a new log
I20260926 21:38:56.188616 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:56.169367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.0.97.62:45107)
0926 21:38:56.170132 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:56.170145 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:56.170196 (+ 51us) server_negotiation.cc:407] Connection header received
0926 21:38:56.171836 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:56.171858 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:56.172271 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:56.172720 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:56.174043 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:56.175375 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:56.176609 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:56.177352 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:56.177539 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:56.181638 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:56.181647 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:56.181662 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:56.181703 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:56.184709 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:56.185495 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:56.185501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:56.185506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:56.185659 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:56.186046 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:56.186049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:56.186051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:56.186517 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:56.186738 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:56.187234 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 21:38:56.187599 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":345,"threads_started":1}
I20260926 21:38:56.191208 11727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d40f3b0d3e49d6a21e1c98f250a892" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45107 } }
I20260926 21:38:56.191581 11727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:56.191682 11727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d40f3b0d3e49d6a21e1c98f250a892, State: Initialized, Role: FOLLOWER
I20260926 21:38:56.192219 11727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d40f3b0d3e49d6a21e1c98f250a892" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45107 } }
I20260926 21:38:56.192466 11727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:38:56.192603 11727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:38:56.192777 11727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:56.194972 11727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d40f3b0d3e49d6a21e1c98f250a892" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45107 } }
I20260926 21:38:56.195498 11727 leader_election.cc:302] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92d40f3b0d3e49d6a21e1c98f250a892; no voters:
I20260926 21:38:56.196079 11727 leader_election.cc:288] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:38:56.196220 11732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:38:56.196957 11732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [term 1 LEADER]: Becoming Leader. State: Replica: 92d40f3b0d3e49d6a21e1c98f250a892, State: Running, Role: LEADER
I20260926 21:38:56.197494 11732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d40f3b0d3e49d6a21e1c98f250a892" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45107 } }
I20260926 21:38:56.198180 11727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:38:56.202327 11734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92d40f3b0d3e49d6a21e1c98f250a892. Latest consensus state: current_term: 1 leader_uuid: "92d40f3b0d3e49d6a21e1c98f250a892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d40f3b0d3e49d6a21e1c98f250a892" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45107 } } }
I20260926 21:38:56.202917 11734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [sys.catalog]: This master's current role is: LEADER
I20260926 21:38:56.203924 11733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92d40f3b0d3e49d6a21e1c98f250a892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d40f3b0d3e49d6a21e1c98f250a892" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45107 } } }
I20260926 21:38:56.204800 11733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892 [sys.catalog]: This master's current role is: LEADER
I20260926 21:38:56.205315 11741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:38:56.211073 11741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:38:56.211705 11727 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 21:38:56.230734 11741 catalog_manager.cc:1347] Generated new cluster ID: 4c4b0680ad164133ba0d8cd3a9c1274e
I20260926 21:38:56.230997 11741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:38:56.245517 11741 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:38:56.247608 11741 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:38:56.256603 11741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892: Generated new TSK 0
I20260926 21:38:56.257833 11741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:38:57.038682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45107
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:38:57 dist-test-slave-lhhq krb5kdc[11216](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 21:38:57.459810 11772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:57.460246 11772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:57.460457 11772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:57.472457 11772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:57.472640 11772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:57.472841 11772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:38:57.488385 11772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:58171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:57.490427 11772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:57.492266 11772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:57.511415 11783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:57.511538 11782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:57.517810 11785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:57.517954 11772 server_base.cc:1034] running on GCE node
I20260926 21:38:57.518731 11772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:57.519806 11772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:57.521061 11772 hybrid_clock.cc:648] HybridClock initialized: now 1790458737520993 us; error 58 us; skew 500 ppm
I20260926 21:38:57.521497 11772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:38:57 dist-test-slave-lhhq krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:38:57.525946 11772 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:38:57.528005 11772 webserver.cc:460] Webserver started at http://127.0.97.1:46327/ using document root <none> and password file <none>
I20260926 21:38:57.529461 11772 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:57.529608 11772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:57.530021 11772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:38:57 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:38:57.550228 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "18597293aa85492a984caba6e65e14b5"
format_stamp: "Formatted at 2026-09-26 21:38:57 on dist-test-slave-lhhq"
server_key: "16cb422d910024789d98a81455033d7e"
server_key_iv: "8db0053664270055df2421d7a28a8a66"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:57.551355 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "18597293aa85492a984caba6e65e14b5"
format_stamp: "Formatted at 2026-09-26 21:38:57 on dist-test-slave-lhhq"
server_key: "16cb422d910024789d98a81455033d7e"
server_key_iv: "8db0053664270055df2421d7a28a8a66"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:57.590735 11772 fs_manager.cc:691] Time spent creating directory manager: real 0.039s user 0.025s sys 0.012s
I20260926 21:38:57.617908 11792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:57.619807 11772 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.007s sys 0.004s
I20260926 21:38:57.620046 11772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "18597293aa85492a984caba6e65e14b5"
format_stamp: "Formatted at 2026-09-26 21:38:57 on dist-test-slave-lhhq"
server_key: "16cb422d910024789d98a81455033d7e"
server_key_iv: "8db0053664270055df2421d7a28a8a66"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:57.620416 11772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:57.692996 11772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:57.698549 11772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:57.698962 11772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:57.706435 11772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:57.714896 11772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:57.715090 11772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:57.715317 11772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:57.715456 11772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:38:57 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:38:58 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:38:58.322968 11772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36071
I20260926 21:38:58.323621 11905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36071 every 8 connection(s)
I20260926 21:38:58.324815 11772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:38:58.335073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 11772
I20260926 21:38:58.335268 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:38:58.336016 388 external_mini_cluster.cc:1428] Setting key 3ce16807bb2a0e52b7b2823e7f291754
Sep 26 21:38:58 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:38:58.366397 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:58.330590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45107 (local address 127.0.97.1:47161)
0926 21:38:58.343565 (+ 12975us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:58.343623 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:58.345669 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:58.346824 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:58.346837 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:58.347469 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:58.348389 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:58.348429 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:58.349957 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:58.349963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:58.351865 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:58.351882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:58.353572 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:58.353581 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:58.353900 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:58.354741 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:58.354771 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:58.357507 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:58.361564 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:58.361574 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:58.361595 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:58.362203 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:58.362676 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:58.362681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:58.362685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:58.362927 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:58.363638 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:58.363650 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:58.364088 (+ 438us) client_negotiation.cc:769] Sending connection context
0926 21:38:58.364412 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 21:38:58.364967 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12638,"thread_start_us":11969,"threads_started":1}
I20260926 21:38:58.366871 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:58.331256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47161 (local address 127.0.97.62:45107)
0926 21:38:58.342482 (+ 11226us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:58.342492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:58.345296 (+ 2804us) server_negotiation.cc:407] Connection header received
0926 21:38:58.346176 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:58.346181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:58.346265 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:58.346364 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:58.348621 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:58.349751 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:58.352035 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:58.352840 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:58.352990 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:58.357807 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:58.357814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:58.357819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:58.357859 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:58.361379 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:58.362363 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:58.362369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:58.362372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:58.362534 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:58.363055 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:58.363060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:58.363063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:58.363411 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:58.363546 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:58.365510 (+ 1964us) server_negotiation.cc:299] Negotiation successful
0926 21:38:58.365757 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11048,"thread_start_us":10922,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:38:58.368621 11906 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45107
I20260926 21:38:58.368901 11906 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:58.369982 11906 heartbeater.cc:511] Master 127.0.97.62:45107 requested a full tablet report, sending...
I20260926 21:38:58.372664 11692 ts_manager.cc:194] Registered new tserver with Master: 18597293aa85492a984caba6e65e14b5 (127.0.97.1:36071)
I20260926 21:38:58.375162 11692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:38:58.401111 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45107
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:38:58.818346 11913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:38:58.818867 11913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:58.819096 11913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:58.827354 11913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:58.827520 11913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:58.827694 11913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:38:58.837620 11913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:58171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:58.839195 11913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:58.840495 11913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:58.855219 11926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:58.855322 11928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:58.856014 11925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:58.856055 11913 server_base.cc:1034] running on GCE node
I20260926 21:38:58.856738 11913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:58.857836 11913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:58.859051 11913 hybrid_clock.cc:648] HybridClock initialized: now 1790458738859005 us; error 43 us; skew 500 ppm
I20260926 21:38:58.859369 11913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:38:58 dist-test-slave-lhhq krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:38:58.862980 11913 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:38:58.864550 11913 webserver.cc:460] Webserver started at http://127.0.97.2:33935/ using document root <none> and password file <none>
I20260926 21:38:58.865293 11913 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:58.865419 11913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:58.865743 11913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:38:58 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:38:58.884117 11913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "4caf90dfa59e4e4188e7254943671eef"
format_stamp: "Formatted at 2026-09-26 21:38:58 on dist-test-slave-lhhq"
server_key: "9d99021deffbca2c464a129946e81d31"
server_key_iv: "b0f064a34bd364c71c0e847b56945ecb"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:58.885224 11913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4caf90dfa59e4e4188e7254943671eef"
format_stamp: "Formatted at 2026-09-26 21:38:58 on dist-test-slave-lhhq"
server_key: "9d99021deffbca2c464a129946e81d31"
server_key_iv: "b0f064a34bd364c71c0e847b56945ecb"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:58.891649 11913 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260926 21:38:58.910074 11935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:58.912034 11913 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:38:58.912230 11913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "4caf90dfa59e4e4188e7254943671eef"
format_stamp: "Formatted at 2026-09-26 21:38:58 on dist-test-slave-lhhq"
server_key: "9d99021deffbca2c464a129946e81d31"
server_key_iv: "b0f064a34bd364c71c0e847b56945ecb"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:58.912560 11913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:58.934481 11913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:58.938807 11913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:58.939117 11913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:58.940083 11913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:58.941509 11913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:58.941608 11913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:58.941762 11913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:58.941835 11913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:58.976857 11913 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33485
I20260926 21:38:58.976892 12047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33485 every 8 connection(s)
I20260926 21:38:58.978277 11913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:38:58.985749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 11913
I20260926 21:38:58.985908 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:38:58.986335 388 external_mini_cluster.cc:1428] Setting key b7b32837c5d1e0066c6038b36cc2371b
Sep 26 21:38:58 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:38:58.999714 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:58.980788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48933 (local address 127.0.97.62:45107)
0926 21:38:58.981311 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:58.981316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:58.982114 (+ 798us) server_negotiation.cc:407] Connection header received
0926 21:38:58.983910 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:58.983916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:58.984004 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:58.984119 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:58.985745 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:58.986643 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:58.988650 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:58.989100 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:58.989249 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:58.993879 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:58.993884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:58.993887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:58.993918 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:58.996491 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:58.997274 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:58.997278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:58.997281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:58.997398 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:58.997856 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:58.997861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:58.997864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:58.998139 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:58.998331 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:58.999075 (+ 744us) server_negotiation.cc:299] Negotiation successful
0926 21:38:58.999311 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":257,"threads_started":1}
I20260926 21:38:59.000672 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:58.981284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45107 (local address 127.0.97.2:48933)
0926 21:38:58.981956 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:58.981995 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:58.983627 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:58.984368 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:58.984376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:58.984814 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:58.985532 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:58.985558 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:58.986829 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:58.986834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:58.988508 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:58.988522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:58.989359 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:58.989369 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:58.989714 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:58.990521 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:58.990554 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:58.993642 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:58.996637 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:58.996647 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:58.996665 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:58.997093 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:58.997510 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:58.997515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:58.997519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:58.997735 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:58.998288 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:58.998296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:58.998702 (+ 406us) client_negotiation.cc:769] Sending connection context
0926 21:38:58.998970 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 21:38:58.999513 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":263,"threads_started":1}
I20260926 21:38:59.002316 12048 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45107
I20260926 21:38:59.002621 12048 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:59.003352 12048 heartbeater.cc:511] Master 127.0.97.62:45107 requested a full tablet report, sending...
I20260926 21:38:59.004770 11692 ts_manager.cc:194] Registered new tserver with Master: 4caf90dfa59e4e4188e7254943671eef (127.0.97.2:33485)
I20260926 21:38:59.005664 11692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48933
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:38:59.039963 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45107
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:38:59.379859 12054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260926 21:38:59.380080 11906 heartbeater.cc:503] Master 127.0.97.62:45107 was elected leader, sending a full tablet report...
W20260926 21:38:59.380219 12054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:38:59.380383 12054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:38:59.388350 12054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:38:59.388507 12054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:38:59.388688 12054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:38:59.398861 12054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:58171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:38:59.400336 12054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:38:59.401772 12054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:38:59.416273 12059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:59.416379 12062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:38:59.416301 12060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:38:59.417225 12054 server_base.cc:1034] running on GCE node
I20260926 21:38:59.417934 12054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:38:59.419114 12054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:38:59.420356 12054 hybrid_clock.cc:648] HybridClock initialized: now 1790458739420302 us; error 51 us; skew 500 ppm
I20260926 21:38:59.420694 12054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:38:59 dist-test-slave-lhhq krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:38:59.423995 12054 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:38:59.426337 12054 webserver.cc:460] Webserver started at http://127.0.97.3:42229/ using document root <none> and password file <none>
I20260926 21:38:59.427140 12054 fs_manager.cc:359] Metadata directory not provided
I20260926 21:38:59.427294 12054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:38:59.427609 12054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:38:59 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:38:59.443825 12054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "604a17fe2e0b424ea6ea82d27ac3d277"
format_stamp: "Formatted at 2026-09-26 21:38:59 on dist-test-slave-lhhq"
server_key: "14565687e3f249c3f3f96119155a8653"
server_key_iv: "b2e5ab00a098a5ea48f6cdc2f0aad197"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:59.444726 12054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "604a17fe2e0b424ea6ea82d27ac3d277"
format_stamp: "Formatted at 2026-09-26 21:38:59 on dist-test-slave-lhhq"
server_key: "14565687e3f249c3f3f96119155a8653"
server_key_iv: "b2e5ab00a098a5ea48f6cdc2f0aad197"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:59.450361 12054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:38:59.467574 12069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:59.469501 12054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 21:38:59.469712 12054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "604a17fe2e0b424ea6ea82d27ac3d277"
format_stamp: "Formatted at 2026-09-26 21:38:59 on dist-test-slave-lhhq"
server_key: "14565687e3f249c3f3f96119155a8653"
server_key_iv: "b2e5ab00a098a5ea48f6cdc2f0aad197"
server_key_version: "kuduclusterkey@0"
I20260926 21:38:59.469988 12054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:38:59.488819 12054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:38:59.492733 12054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:38:59.493009 12054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:38:59.494102 12054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:38:59.495474 12054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:38:59.495549 12054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:59.495636 12054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:38:59.495682 12054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:38:59.532718 12054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37653
I20260926 21:38:59.532756 12181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37653 every 8 connection(s)
I20260926 21:38:59.534544 12054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:38:59.543515 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 12054
I20260926 21:38:59.543702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:38:59.544234 388 external_mini_cluster.cc:1428] Setting key 3e7c7cadc9d863e9d9d34b333f70ac79
Sep 26 21:38:59 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:38:59.554824 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.537156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45571 (local address 127.0.97.62:45107)
0926 21:38:59.537648 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:59.537652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:59.538576 (+ 924us) server_negotiation.cc:407] Connection header received
0926 21:38:59.540358 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:59.540363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:59.540446 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:59.540572 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:59.542072 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.542831 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.545086 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.545805 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.545908 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.549624 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:59.549629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:59.549633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:59.549666 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:59.552116 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:59.552834 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:59.552837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:59.552839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:59.552951 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:59.553403 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:59.553405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:59.553407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:59.553660 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:59.553764 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:59.554404 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 21:38:59.554570 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":223,"threads_started":1}
I20260926 21:38:59.556300 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.537733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45107 (local address 127.0.97.3:45571)
0926 21:38:59.538409 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:59.538447 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:59.540083 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:59.540823 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:59.540830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:59.541322 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:38:59.541905 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.541922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.542974 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.542981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.544917 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.544929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.546045 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.546054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.546327 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.547078 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:38:59.547126 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:38:59.549408 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:38:59.552248 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:59.552255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:59.552269 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:59.552731 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:59.553040 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:38:59.553043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:38:59.553045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:38:59.553303 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:38:59.553769 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:38:59.553776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:38:59.554087 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 21:38:59.554312 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 21:38:59.554714 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":270,"threads_started":1}
I20260926 21:38:59.557729 12182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45107
I20260926 21:38:59.558073 12182 heartbeater.cc:464] Registering TS with master...
I20260926 21:38:59.558976 12182 heartbeater.cc:511] Master 127.0.97.62:45107 requested a full tablet report, sending...
I20260926 21:38:59.560603 11692 ts_manager.cc:194] Registered new tserver with Master: 604a17fe2e0b424ea6ea82d27ac3d277 (127.0.97.3:37653)
I20260926 21:38:59.561486 11692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45571
I20260926 21:38:59.572386 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:38:59.589809 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.576954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.0.97.62:45107)
0926 21:38:59.577163 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:59.577180 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:59.577586 (+ 406us) server_negotiation.cc:407] Connection header received
0926 21:38:59.577794 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:59.577800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:59.577883 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:59.577984 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:59.579424 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.580145 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.581596 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.582000 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.582131 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.584347 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:59.584353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:59.584359 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:59.584396 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:59.587157 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:59.587872 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:59.587880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:59.587884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:59.588019 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:59.588405 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:59.588411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:59.588414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:59.588675 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:59.588918 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:59.589263 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 21:38:59.589480 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:38:59.593932 11692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:38:59.601027 11692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:38:59.627465 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.615586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.0.97.2:33485)
0926 21:38:59.616461 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:59.616472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:59.617253 (+ 781us) server_negotiation.cc:407] Connection header received
0926 21:38:59.617685 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:59.617695 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:59.617988 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:59.618150 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:59.619278 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.620920 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.623889 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.625796 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.625908 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.626123 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:59.626944 (+ 821us) server_negotiation.cc:299] Negotiation successful
0926 21:38:59.627153 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":593,"threads_started":1}
I20260926 21:38:59.627519 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.615604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36071 (local address 127.0.0.1:47026)
0926 21:38:59.617759 (+ 2155us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:59.617776 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:59.617896 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:59.618642 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:59.618648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:59.618676 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:59.619700 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.619713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.621895 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.621899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.623999 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.624011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.626463 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.626469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.626618 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.626641 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:38:59.626726 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 21:38:59.627016 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2028,"mutex_wait_us":1,"thread_start_us":2557,"threads_started":1}
I20260926 21:38:59.628373 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.615496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33485 (local address 127.0.0.1:53116)
0926 21:38:59.617051 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:59.617104 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:59.617412 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:59.618462 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:59.618471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:59.618504 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:59.619012 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.619032 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.621102 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.621109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.623707 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.623721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.626317 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.626323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.626539 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.626593 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 21:38:59.627846 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0926 21:38:59.627982 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1321,"mutex_wait_us":30,"thread_start_us":1143,"threads_started":1}
I20260926 21:38:59.628659 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.616149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.0.97.1:36071)
0926 21:38:59.617743 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:59.617751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:59.617846 (+ 95us) server_negotiation.cc:407] Connection header received
0926 21:38:59.618061 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:59.618070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:59.618343 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:59.618740 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:59.619943 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.621700 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.624169 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.625988 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.626098 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.626328 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:59.628098 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0926 21:38:59.628293 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1425,"thread_start_us":330,"threads_started":1}
I20260926 21:38:59.633390 11983 tablet_service.cc:1467] Processing CreateTablet for tablet a2ccbfb2595e470cb74a3d448445893e (DEFAULT_TABLE table=test-table [id=fbaee868d02c4bafb82841611e241277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:38:59.635020 11983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ccbfb2595e470cb74a3d448445893e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:59.634684 11840 tablet_service.cc:1467] Processing CreateTablet for tablet a2ccbfb2595e470cb74a3d448445893e (DEFAULT_TABLE table=test-table [id=fbaee868d02c4bafb82841611e241277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:38:59.636166 11840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ccbfb2595e470cb74a3d448445893e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:59.636910 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.618681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37653 (local address 127.0.0.1:55090)
0926 21:38:59.627821 (+ 9140us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:59.627840 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:59.627939 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:59.631386 (+ 3447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:59.631391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:59.631420 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:59.631577 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.631587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.632931 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.632938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.634636 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.634665 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.636063 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.636069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.636236 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.636315 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 21:38:59.636441 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 21:38:59.636586 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8987,"thread_start_us":12010,"threads_started":1}
I20260926 21:38:59.636965 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.620909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.0.97.3:37653)
0926 21:38:59.630699 (+ 9790us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:59.630712 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:59.630754 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:38:59.630857 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:59.630867 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:59.631170 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:59.631383 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:59.631740 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.632794 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.634807 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.635903 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.636051 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.636300 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:59.636480 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 21:38:59.636631 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9592,"thread_start_us":9429,"threads_started":1}
I20260926 21:38:59.641042 12117 tablet_service.cc:1467] Processing CreateTablet for tablet a2ccbfb2595e470cb74a3d448445893e (DEFAULT_TABLE table=test-table [id=fbaee868d02c4bafb82841611e241277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:38:59.642386 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ccbfb2595e470cb74a3d448445893e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:38:59.651130 12202 tablet_bootstrap.cc:492] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: Bootstrap starting.
I20260926 21:38:59.652292 12201 tablet_bootstrap.cc:492] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: Bootstrap starting.
I20260926 21:38:59.654631 12203 tablet_bootstrap.cc:492] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: Bootstrap starting.
I20260926 21:38:59.655938 12201 tablet_bootstrap.cc:654] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:59.656978 12202 tablet_bootstrap.cc:654] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:59.657325 12201 log.cc:824] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: Log is configured to *not* fsync() on all Append() calls
I20260926 21:38:59.658238 12202 log.cc:824] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: Log is configured to *not* fsync() on all Append() calls
I20260926 21:38:59.658474 12203 tablet_bootstrap.cc:654] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: Neither blocks nor log segments found. Creating new log.
I20260926 21:38:59.659783 12203 log.cc:824] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: Log is configured to *not* fsync() on all Append() calls
I20260926 21:38:59.660095 12201 tablet_bootstrap.cc:492] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: No bootstrap required, opened a new log
I20260926 21:38:59.660391 12201 ts_tablet_manager.cc:1397] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 21:38:59.663120 12203 tablet_bootstrap.cc:492] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: No bootstrap required, opened a new log
I20260926 21:38:59.663125 12202 tablet_bootstrap.cc:492] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: No bootstrap required, opened a new log
I20260926 21:38:59.663378 12203 ts_tablet_manager.cc:1397] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 21:38:59.663378 12202 ts_tablet_manager.cc:1397] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 21:38:59.668373 12201 raft_consensus.cc:348] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } }
I20260926 21:38:59.669126 12201 raft_consensus.cc:374] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:59.669308 12201 raft_consensus.cc:729] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4caf90dfa59e4e4188e7254943671eef, State: Initialized, Role: FOLLOWER
I20260926 21:38:59.669947 12201 consensus_queue.cc:260] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } }
I20260926 21:38:59.671041 12048 heartbeater.cc:503] Master 127.0.97.62:45107 was elected leader, sending a full tablet report...
I20260926 21:38:59.671818 12201 ts_tablet_manager.cc:1428] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260926 21:38:59.674391 12202 raft_consensus.cc:348] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } }
I20260926 21:38:59.674402 12203 raft_consensus.cc:348] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } }
I20260926 21:38:59.674853 12202 raft_consensus.cc:374] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:59.674855 12203 raft_consensus.cc:374] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:38:59.674954 12203 raft_consensus.cc:729] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604a17fe2e0b424ea6ea82d27ac3d277, State: Initialized, Role: FOLLOWER
I20260926 21:38:59.675035 12202 raft_consensus.cc:729] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18597293aa85492a984caba6e65e14b5, State: Initialized, Role: FOLLOWER
I20260926 21:38:59.675585 12202 consensus_queue.cc:260] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } }
I20260926 21:38:59.675604 12203 consensus_queue.cc:260] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } }
I20260926 21:38:59.676707 12182 heartbeater.cc:503] Master 127.0.97.62:45107 was elected leader, sending a full tablet report...
I20260926 21:38:59.677491 12203 ts_tablet_manager.cc:1428] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: Time spent starting tablet: real 0.014s user 0.005s sys 0.011s
I20260926 21:38:59.677775 12202 ts_tablet_manager.cc:1428] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260926 21:38:59.730310 12049 tablet.cc:2369] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:38:59.731029 12209 raft_consensus.cc:482] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:38:59.731418 12209 raft_consensus.cc:504] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } }
I20260926 21:38:59.733810 12209 leader_election.cc:288] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4caf90dfa59e4e4188e7254943671eef (127.0.97.2:33485), 604a17fe2e0b424ea6ea82d27ac3d277 (127.0.97.3:37653)
I20260926 21:38:59.742322 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.735246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56469 (local address 127.0.97.3:37653)
0926 21:38:59.735452 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:59.735458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:59.735862 (+ 404us) server_negotiation.cc:407] Connection header received
0926 21:38:59.736157 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:59.736166 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:59.736275 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:59.736380 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:59.737366 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.738090 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.739879 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.741359 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.741509 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.741613 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:59.741945 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 21:38:59.742065 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:38:59.742502 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.734224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37653 (local address 127.0.97.1:56469)
0926 21:38:59.735749 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:59.735780 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:59.735985 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:59.736553 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:59.736558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:59.736580 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:59.737145 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.737163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.738227 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.738235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.739717 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.739730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.741563 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.741569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.741737 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.741770 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:38:59.741917 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 21:38:59.742054 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"mutex_wait_us":51,"thread_start_us":1235,"threads_started":1}
I20260926 21:38:59.742600 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.735828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58563 (local address 127.0.97.2:33485)
0926 21:38:59.736023 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:59.736028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:59.736046 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:38:59.736157 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:59.736163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:59.736259 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:59.736361 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:38:59.737386 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.738449 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.740373 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.741495 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.741632 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.741736 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:59.742210 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 21:38:59.742333 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:38:59.742775 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.734064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33485 (local address 127.0.97.1:58563)
0926 21:38:59.734857 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 21:38:59.734891 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 21:38:59.735075 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:38:59.736390 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:38:59.736394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:38:59.736417 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:38:59.737155 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.737174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.738584 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.738591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.740162 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:38:59.740177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.741902 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.741908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:38:59.742054 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.742076 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:38:59.742187 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 21:38:59.742333 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":140,"thread_start_us":380,"threads_started":1}
I20260926 21:38:59.743649 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ccbfb2595e470cb74a3d448445893e" candidate_uuid: "18597293aa85492a984caba6e65e14b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" is_pre_election: true
I20260926 21:38:59.744279 12137 raft_consensus.cc:2454] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18597293aa85492a984caba6e65e14b5 in term 0.
I20260926 21:38:59.744246 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ccbfb2595e470cb74a3d448445893e" candidate_uuid: "18597293aa85492a984caba6e65e14b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4caf90dfa59e4e4188e7254943671eef" is_pre_election: true
I20260926 21:38:59.744781 12003 raft_consensus.cc:2454] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18597293aa85492a984caba6e65e14b5 in term 0.
I20260926 21:38:59.745013 11794 leader_election.cc:302] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18597293aa85492a984caba6e65e14b5, 604a17fe2e0b424ea6ea82d27ac3d277; no voters:
I20260926 21:38:59.745482 12209 raft_consensus.cc:2781] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:38:59.745627 12209 raft_consensus.cc:482] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:38:59.745724 12209 raft_consensus.cc:3037] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:59.747852 12209 raft_consensus.cc:504] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } }
I20260926 21:38:59.748426 12209 leader_election.cc:288] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [CANDIDATE]: Term 1 election: Requested vote from peers 4caf90dfa59e4e4188e7254943671eef (127.0.97.2:33485), 604a17fe2e0b424ea6ea82d27ac3d277 (127.0.97.3:37653)
I20260926 21:38:59.748873 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ccbfb2595e470cb74a3d448445893e" candidate_uuid: "18597293aa85492a984caba6e65e14b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4caf90dfa59e4e4188e7254943671eef"
I20260926 21:38:59.748948 12137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ccbfb2595e470cb74a3d448445893e" candidate_uuid: "18597293aa85492a984caba6e65e14b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a17fe2e0b424ea6ea82d27ac3d277"
I20260926 21:38:59.749082 12003 raft_consensus.cc:3037] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:59.749123 12137 raft_consensus.cc:3037] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:38:59.752308 12003 raft_consensus.cc:2454] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18597293aa85492a984caba6e65e14b5 in term 1.
I20260926 21:38:59.752303 12137 raft_consensus.cc:2454] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18597293aa85492a984caba6e65e14b5 in term 1.
I20260926 21:38:59.752712 11794 leader_election.cc:302] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18597293aa85492a984caba6e65e14b5, 604a17fe2e0b424ea6ea82d27ac3d277; no voters:
I20260926 21:38:59.753054 12209 raft_consensus.cc:2781] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:38:59.753562 12209 raft_consensus.cc:686] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 1 LEADER]: Becoming Leader. State: Replica: 18597293aa85492a984caba6e65e14b5, State: Running, Role: LEADER
I20260926 21:38:59.754046 12209 consensus_queue.cc:237] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } }
I20260926 21:38:59.759322 11692 catalog_manager.cc:5495] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18597293aa85492a984caba6e65e14b5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "18597293aa85492a984caba6e65e14b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18597293aa85492a984caba6e65e14b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 } health_report { overall_health: UNKNOWN } } }
I20260926 21:38:59.784338 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.778724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.0.97.1:36071)
0926 21:38:59.778931 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:59.778939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:59.778959 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:38:59.779222 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:59.779226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:59.779322 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:59.779476 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:38:59.779868 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.780574 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.782158 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.782558 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.782696 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.783204 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:38:59.783716 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:38:59.783874 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:59.783985 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 21:38:59.784063 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 21:38:59.786128 12183 tablet.cc:2369] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:38:59.793215 12003 raft_consensus.cc:1261] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [term 1 FOLLOWER]: Refusing update from remote peer 18597293aa85492a984caba6e65e14b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:38:59.793182 12137 raft_consensus.cc:1261] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [term 1 FOLLOWER]: Refusing update from remote peer 18597293aa85492a984caba6e65e14b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:38:59.794503 12209 consensus_queue.cc:1035] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4caf90dfa59e4e4188e7254943671eef" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:38:59.794893 12212 consensus_queue.cc:1035] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604a17fe2e0b424ea6ea82d27ac3d277" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:38:59.827175 11692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:38:59.827471 11692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:38:59.834051 11692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92d40f3b0d3e49d6a21e1c98f250a892: Sending DeleteTablet for 3 replicas of tablet a2ccbfb2595e470cb74a3d448445893e
I20260926 21:38:59.835858 11983 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ccbfb2595e470cb74a3d448445893e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:38:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53116
I20260926 21:38:59.836056 11840 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ccbfb2595e470cb74a3d448445893e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:38:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47026
I20260926 21:38:59.836135 12117 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ccbfb2595e470cb74a3d448445893e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:38:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55090
I20260926 21:38:59.836645 12116 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ccbfb2595e470cb74a3d448445893e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:38:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55090
I20260926 21:38:59.836750 11839 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ccbfb2595e470cb74a3d448445893e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:38:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47026
I20260926 21:38:59.840452 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 11772
I20260926 21:38:59.850889 12226 tablet_replica.cc:330] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: stopping tablet replica
I20260926 21:38:59.852347 12226 raft_consensus.cc:2229] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:38:59.854180 12226 raft_consensus.cc:2258] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:38:59.857295 12227 tablet_replica.cc:330] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: stopping tablet replica
I20260926 21:38:59.857883 12227 raft_consensus.cc:2229] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:38:59.858500 12227 raft_consensus.cc:2258] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:38:59.860924 12227 ts_tablet_manager.cc:1905] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:38:59.861361 12226 ts_tablet_manager.cc:1905] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:38:59.862002 12225 tablet_replica.cc:330] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: stopping tablet replica
I20260926 21:38:59.862389 12225 raft_consensus.cc:2229] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:38:59.862823 12225 raft_consensus.cc:2258] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:38:59.865675 12225 ts_tablet_manager.cc:1905] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:38:59.868091 12227 ts_tablet_manager.cc:1918] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:38:59.868315 12227 log.cc:1196] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/a2ccbfb2595e470cb74a3d448445893e
I20260926 21:38:59.868891 12227 ts_tablet_manager.cc:1939] T a2ccbfb2595e470cb74a3d448445893e P 604a17fe2e0b424ea6ea82d27ac3d277: Deleting consensus metadata
I20260926 21:38:59.871322 11677 catalog_manager.cc:4841] TS 604a17fe2e0b424ea6ea82d27ac3d277 (127.0.97.3:37653): tablet a2ccbfb2595e470cb74a3d448445893e (table test-table [id=fbaee868d02c4bafb82841611e241277]) successfully deleted
I20260926 21:38:59.872442 12225 ts_tablet_manager.cc:1918] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 21:38:59.872428 11677 catalog_manager.cc:4808] TS 604a17fe2e0b424ea6ea82d27ac3d277 (127.0.97.3:37653): delete failed for tablet a2ccbfb2595e470cb74a3d448445893e because the tablet was not found. No further retry: Not found: Tablet not found: a2ccbfb2595e470cb74a3d448445893e
I20260926 21:38:59.872627 12225 log.cc:1196] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/a2ccbfb2595e470cb74a3d448445893e
I20260926 21:38:59.873142 12225 ts_tablet_manager.cc:1939] T a2ccbfb2595e470cb74a3d448445893e P 4caf90dfa59e4e4188e7254943671eef: Deleting consensus metadata
I20260926 21:38:59.874199 12226 ts_tablet_manager.cc:1918] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:38:59.874382 12226 log.cc:1196] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/a2ccbfb2595e470cb74a3d448445893e
I20260926 21:38:59.874985 12226 ts_tablet_manager.cc:1939] T a2ccbfb2595e470cb74a3d448445893e P 18597293aa85492a984caba6e65e14b5: Deleting consensus metadata
I20260926 21:38:59.874950 11679 catalog_manager.cc:4841] TS 4caf90dfa59e4e4188e7254943671eef (127.0.97.2:33485): tablet a2ccbfb2595e470cb74a3d448445893e (table test-table [id=fbaee868d02c4bafb82841611e241277]) successfully deleted
I20260926 21:38:59.876510 11677 catalog_manager.cc:4841] TS 18597293aa85492a984caba6e65e14b5 (127.0.97.1:36071): tablet a2ccbfb2595e470cb74a3d448445893e (table test-table [id=fbaee868d02c4bafb82841611e241277]) successfully deleted
W20260926 21:38:59.876937 11677 catalog_manager.cc:4808] TS 18597293aa85492a984caba6e65e14b5 (127.0.97.1:36071): delete failed for tablet a2ccbfb2595e470cb74a3d448445893e because the tablet was not found. No further retry: Not found: Tablet not found: a2ccbfb2595e470cb74a3d448445893e
Sep 26 21:38:59 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:38:59.888386 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:38:59.841967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.0.97.1:36071)
0926 21:38:59.842145 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 21:38:59.842151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:38:59.851766 (+ 9615us) server_negotiation.cc:407] Connection header received
0926 21:38:59.852986 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:38:59.852995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:38:59.853104 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:38:59.853233 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:38:59.863255 (+ 10022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.864086 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.876944 (+ 12858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:38:59.877535 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:38:59.877622 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:38:59.882270 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:38:59.882282 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:38:59.882297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:38:59.882342 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:38:59.885512 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:59.886246 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:59.886253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:59.886255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:59.886414 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:38:59.886823 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:38:59.886827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:38:59.886829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:38:59.887183 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:38:59.887483 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 21:38:59.887809 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 21:38:59.888049 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:38:59.889132 11901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:38:59.990604 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 11772
I20260926 21:39:00.009760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 11913
Sep 26 21:39:00 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:39:00.024819 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:00.010557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.0.97.2:33485)
0926 21:39:00.010746 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:00.010751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:00.010803 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:39:00.011042 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:00.011047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:00.011151 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:00.011277 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:00.012515 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:00.013321 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:00.014567 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:00.014945 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:00.015043 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:00.019493 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:00.019508 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:00.019523 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:00.019554 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:00.022462 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:00.023091 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:00.023098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:00.023100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:00.023219 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:00.023533 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:00.023536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:00.023538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:00.023873 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:00.024082 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:00.024296 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 21:39:00.024447 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:39:00.025457 12043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:00.115633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 11913
I20260926 21:39:00.132162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 12054
Sep 26 21:39:00 dist-test-slave-lhhq krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:39:00.147099 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:00.132911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.0.97.3:37653)
0926 21:39:00.133070 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:00.133074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:00.133110 (+ 36us) server_negotiation.cc:407] Connection header received
0926 21:39:00.133362 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:00.133366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:00.133432 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:00.133528 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:00.134773 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:00.135491 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:00.136805 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:00.137161 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:00.137463 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:00.141760 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:00.141773 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:00.141789 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:00.141821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:00.144301 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:00.144991 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:00.144999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:00.145001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:00.145113 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:00.145539 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:00.145543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:00.145548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:00.145916 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:00.146197 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:00.146505 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 21:39:00.146715 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:39:00.147678 12177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:00.237344 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 12054
I20260926 21:39:00.253769 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 11660
I20260926 21:39:00.255005 11722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:00.367110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 11660
I20260926 21:39:00.393620 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 21:39:00.729161 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 21:39:00.729435 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 21:39:01.074885 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 21:39:01.075 UTC [11246] LOG: received smart shutdown request
2026-09-26 21:39:01.078 UTC [11246] LOG: background worker "logical replication launcher" (PID 11255) exited with exit code 1
2026-09-26 21:39:01.079 UTC [11250] LOG: shutting down
2026-09-26 21:39:01.240 UTC [11246] LOG: database system is shut down
2026-09-26T21:39:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50309 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:39:01 dist-test-slave-lhhq krb5kdc[12241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:39:01 dist-test-slave-lhhq krb5kdc[12241](info): set up 2 sockets
Sep 26 21:39:01 dist-test-slave-lhhq krb5kdc[12241](info): commencing operation
krb5kdc: starting...
W20260926 21:39:02.489589 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:39:02 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:39:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 21:39:02.824812 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh3AZIa/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/postgres -l logfile start
2026-09-26 21:39:05.358 UTC [12271] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 21:39:05.358 UTC [12271] LOG: listening on IPv4 address "127.0.97.20", port 36347
2026-09-26 21:39:05.361 UTC [12271] LOG: listening on Unix socket "/tmp/.s.PGSQL.36347"
2026-09-26 21:39:05.383 UTC [12274] LOG: database system was shut down at 2026-09-26 21:39:03 UTC
2026-09-26 21:39:05.386 UTC [12271] LOG: database system is ready to accept connections
I20260926 21:39:06.365320 388 mini_postgres.cc:96] Postgres bound to 36347
2026-09-26 21:39:06.382 UTC [12282] FATAL: database "slave" does not exist
127.0.97.20:36347 - accepting connections
I20260926 21:39:06.383541 388 mini_ranger.cc:161] Starting Ranger...
I20260926 21:39:06.415066 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 21:39:06.926923 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 21:39:06.927045 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-admin
2026-09-26 21:39:07,224 [I] DB FLAVOR :POSTGRES
2026-09-26 21:39:07,224 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 21:39:07,225 [I] Checking connection..
2026-09-26 21:39:07,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 21:39:07,476 [I] Checking connection passed.
2026-09-26 21:39:07,476 [I] --------- Verifying version history table ---------
2026-09-26 21:39:07,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:39:07,723 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 21:39:07,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:39:07,992 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 21:39:07,992 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 21:39:07,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 21:39:08.217 UTC [12346] WARNING: there is no transaction in progress
2026-09-26 21:39:08,224 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 21:39:08,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 21:39:08,490 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 21:39:08,491 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 21:39:08,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 21:39:08,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 21:39:08,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lhhq.c.gcp-upstream.internal','N') ;"
2026-09-26 21:39:09,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 21:39:09,496 [I] Table x_portal_user does not exist in database ranger
2026-09-26 21:39:09,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 21:39:09,743 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 21:39:09,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 21:39:09,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 21:39:10,232 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 21:39:10,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 21:39:11.187 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.203 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.219 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.235 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.251 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.268 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.322 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.328 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.343 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.354 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.365 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.380 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.390 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.397 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.405 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:11.412 UTC [12454] WARNING: there is no transaction in progress
2026-09-26 21:39:12,065 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 21:39:12,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 21:39:12,289 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 21:39:12,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 21:39:12,552 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 21:39:12,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 21:39:12,824 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 21:39:12,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 21:39:13,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 21:39:13,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lhhq.c.gcp-upstream.internal';"
2026-09-26 21:39:13,592 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 21:39:13,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 21:39:13,850 [I] DB_PATCHES have already been applied
I20260926 21:39:13.856143 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh3AZIa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 21:39:13.856264 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 21:39:13.870824 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:51957
Sep 26, 2026 9:39:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:39:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:39:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:39:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:39:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:39:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:39:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 21:39:15.006399 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.623s user 0.000s sys 0.043s
Sep 26, 2026 9:39:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:39:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 9:39:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:39:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:39:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 21:39:37.248903 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 21:39:37 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458777, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 21:39:37.431869 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 21:39:37.453977 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 21:39:38.336112 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 21:39:38.336232 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-kms
I20260926 21:39:38.336292 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:36347
2026-09-26 21:39:38,698 [I] DB FLAVOR :POSTGRES
2026-09-26 21:39:38,699 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 21:39:38,699 [I] Checking connection
2026-09-26 21:39:38,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 21:39:38,932 [I] connection success
2026-09-26 21:39:38,932 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 21:39:38,932 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 21:39:38,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 21:39:39,166 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 21:39:39,167 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 21:39:39,167 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 21:39:39,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36347/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 21:39:39,433 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 21:39:39.892797 388 mini_ranger_kms.cc:310] Created kms service
I20260926 21:39:40.091936 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 21:39:40.136063 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 21:39:40 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458777, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:40.539666 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 21:39:40.539986 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh3AZIa/build/asan/bin/postgresql.jar:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh3AZIa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 21:39:40.540045 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 21:39:40.554726 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12634
I20260926 21:39:40.554862 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:52289
Sep 26, 2026 9:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:39:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:39:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 21:39:42 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458782, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:42.872912 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.441s user 0.043s sys 0.335s
I20260926 21:39:42.873158 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 21:39:42.873353 388 mini_ranger_kms.cc:401] 127.0.97.20:52289/kms/v1/keys
Sep 26 21:39:43 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458782, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:39:43 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458782, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:39:45 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:39:45.883038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39981
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39981
--ranger_config_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52289/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:39:46.206111 12685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:46.206408 12685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:46.206561 12685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:46.214078 12685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:39:46.214187 12685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:46.214286 12685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:46.214382 12685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:39:46.214466 12685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:39:46.224997 12685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:52289/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39981
--ranger_config_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:46.226338 12685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:46.227669 12685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:46.239332 12690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:46.239387 12691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:46.239529 12693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:46.240391 12685 server_base.cc:1034] running on GCE node
I20260926 21:39:46.241142 12685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:46.242616 12685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:46.243851 12685 hybrid_clock.cc:648] HybridClock initialized: now 1790458786243809 us; error 56 us; skew 500 ppm
I20260926 21:39:46.244205 12685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:46 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:46.253648 12685 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:39:46.255262 12685 webserver.cc:460] Webserver started at http://127.0.97.62:39817/ using document root <none> and password file <none>
I20260926 21:39:46.255985 12685 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:46.256125 12685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:39:46.256428 12685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:39:46 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:46.306442 12685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "3ac1447611fb47f59194e0c32a4523e4"
format_stamp: "Formatted at 2026-09-26 21:39:46 on dist-test-slave-lhhq"
server_key: "aba2683a95d08f65fcb1571f9403967b"
server_key_iv: "66c49c4f1ceaae37897ca25ce804eb66"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:46.307284 12685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "3ac1447611fb47f59194e0c32a4523e4"
format_stamp: "Formatted at 2026-09-26 21:39:46 on dist-test-slave-lhhq"
server_key: "aba2683a95d08f65fcb1571f9403967b"
server_key_iv: "66c49c4f1ceaae37897ca25ce804eb66"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:46.313028 12685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:39:46.329702 12700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:46.331374 12685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 21:39:46.331588 12685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "3ac1447611fb47f59194e0c32a4523e4"
format_stamp: "Formatted at 2026-09-26 21:39:46 on dist-test-slave-lhhq"
server_key: "aba2683a95d08f65fcb1571f9403967b"
server_key_iv: "66c49c4f1ceaae37897ca25ce804eb66"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:46.331827 12685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:46.354627 12685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:46.358619 12685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:46.358943 12685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:46.378077 12685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39981
I20260926 21:39:46.378118 12751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39981 every 8 connection(s)
I20260926 21:39:46.379513 12685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:39:46.383220 12752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:39:46.387074 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 12685
I20260926 21:39:46.387317 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:39:46.387959 388 external_mini_cluster.cc:1428] Setting key 81884210bffaa54fd69b7d35be29bc51
I20260926 21:39:46.395239 12752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Bootstrap starting.
I20260926 21:39:46.398870 12752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Neither blocks nor log segments found. Creating new log.
I20260926 21:39:46.400122 12752 log.cc:824] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Log is configured to *not* fsync() on all Append() calls
I20260926 21:39:46.403420 12752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: No bootstrap required, opened a new log
Sep 26 21:39:46 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:39:46.413496 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:46.389998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.0.97.62:39981)
0926 21:39:46.390763 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:46.390779 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:46.390825 (+ 46us) server_negotiation.cc:407] Connection header received
0926 21:39:46.393463 (+ 2638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:46.393493 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:46.393974 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:46.394397 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:46.395545 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:46.397584 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:46.398879 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:46.399910 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:46.400090 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:46.404437 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:46.404454 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:46.404472 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:46.404523 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:46.408753 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:46.409549 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:46.409558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:46.409567 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:46.409752 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:46.410114 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:46.410124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:46.410127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:46.410863 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:46.411160 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:46.411501 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 21:39:46.412055 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":307,"threads_started":1}
I20260926 21:39:46.414376 12752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } }
I20260926 21:39:46.414767 12752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:39:46.414885 12752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac1447611fb47f59194e0c32a4523e4, State: Initialized, Role: FOLLOWER
I20260926 21:39:46.415561 12752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } }
I20260926 21:39:46.415839 12752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:39:46.415982 12752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:39:46.416190 12752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:39:46.418627 12752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } }
I20260926 21:39:46.419167 12752 leader_election.cc:302] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ac1447611fb47f59194e0c32a4523e4; no voters:
I20260926 21:39:46.419677 12752 leader_election.cc:288] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:39:46.419833 12757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:39:46.421092 12757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 1 LEADER]: Becoming Leader. State: Replica: 3ac1447611fb47f59194e0c32a4523e4, State: Running, Role: LEADER
I20260926 21:39:46.421768 12757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } }
I20260926 21:39:46.421926 12752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:39:46.424783 12759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ac1447611fb47f59194e0c32a4523e4. Latest consensus state: current_term: 1 leader_uuid: "3ac1447611fb47f59194e0c32a4523e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } } }
I20260926 21:39:46.425103 12758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ac1447611fb47f59194e0c32a4523e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } } }
I20260926 21:39:46.425685 12759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: This master's current role is: LEADER
I20260926 21:39:46.425685 12758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: This master's current role is: LEADER
I20260926 21:39:46.427621 12766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:39:46.432255 12766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:39:46.432989 12752 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 21:39:46.451151 12766 catalog_manager.cc:1347] Generated new cluster ID: 297c425b54a94fe58b76848ccd050be3
I20260926 21:39:46.451308 12766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:39:46.469076 12766 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:39:46.470677 12766 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:39:46.478140 12766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Generated new TSK 0
I20260926 21:39:46.479075 12766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:39:47.246102 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39981
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:39:47 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 21:39:47.670601 12797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:47.670953 12797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:47.671128 12797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:47.679184 12797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:47.679342 12797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:47.679512 12797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:39:47.690011 12797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:52289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:47.691524 12797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:47.693053 12797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:47.712678 12807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:47.713392 12808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:47.718827 12797 server_base.cc:1034] running on GCE node
W20260926 21:39:47.719151 12811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:47.720012 12797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:47.729287 12797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:47.733177 12797 hybrid_clock.cc:648] HybridClock initialized: now 1790458787732990 us; error 160 us; skew 500 ppm
I20260926 21:39:47.733630 12797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:47 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:47.737617 12797 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:39:47.750716 12797 webserver.cc:460] Webserver started at http://127.0.97.1:38057/ using document root <none> and password file <none>
I20260926 21:39:47.751538 12797 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:47.751699 12797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:39:47.752048 12797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:39:47 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:47.769645 12797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "270ae1110c174b2399e2e1f79e08050d"
format_stamp: "Formatted at 2026-09-26 21:39:47 on dist-test-slave-lhhq"
server_key: "b335689bc166ee0dc0f8fa0d4fbb8e5b"
server_key_iv: "4382eb595e296701ac75bd3e3049f283"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:47.770884 12797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "270ae1110c174b2399e2e1f79e08050d"
format_stamp: "Formatted at 2026-09-26 21:39:47 on dist-test-slave-lhhq"
server_key: "b335689bc166ee0dc0f8fa0d4fbb8e5b"
server_key_iv: "4382eb595e296701ac75bd3e3049f283"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:47.783098 12797 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.007s
I20260926 21:39:47.829778 12817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:47.831560 12797 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.004s
I20260926 21:39:47.831791 12797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "270ae1110c174b2399e2e1f79e08050d"
format_stamp: "Formatted at 2026-09-26 21:39:47 on dist-test-slave-lhhq"
server_key: "b335689bc166ee0dc0f8fa0d4fbb8e5b"
server_key_iv: "4382eb595e296701ac75bd3e3049f283"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:47.832033 12797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:47.878036 12797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:47.885360 12797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:47.885826 12797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:47.893415 12797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:39:47.901877 12797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:39:47.902086 12797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 21:39:47.902331 12797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:39:47.902482 12797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:39:48 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 21:39:48 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:48.617731 12797 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43911
I20260926 21:39:48.617940 12931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43911 every 8 connection(s)
I20260926 21:39:48.619531 12797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:39:48.621729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 12797
I20260926 21:39:48.621986 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:39:48.623004 388 external_mini_cluster.cc:1428] Setting key 991f42b1eb4cc427ead2d0276591a471
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Sep 26 21:39:48 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 21:39:48.687626 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:48.640840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39981 (local address 127.0.97.1:40351)
0926 21:39:48.655142 (+ 14302us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:48.655216 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:48.660341 (+ 5125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:48.663181 (+ 2840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:48.663195 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:48.663840 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:48.664661 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:48.664687 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:48.666288 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:48.666294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:48.668195 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:48.668211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:48.669449 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:48.669456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:48.669742 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:48.670688 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:48.670726 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:48.673551 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:48.682247 (+ 8696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:48.682266 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:48.682288 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:48.682996 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:48.683558 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:48.683563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:48.683567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:48.683953 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:48.684756 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:48.684766 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:48.685277 (+ 511us) client_negotiation.cc:769] Sending connection context
0926 21:39:48.685625 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 21:39:48.686192 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13876,"thread_start_us":14555,"threads_started":1}
I20260926 21:39:48.688848 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:48.640172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40351 (local address 127.0.97.62:39981)
0926 21:39:48.662312 (+ 22140us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:48.662318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:48.662342 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:39:48.662408 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:48.662413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:48.662500 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:48.662696 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:48.664907 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:48.666079 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:48.668376 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:48.668985 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:48.669110 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:48.673851 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:48.673859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:48.673864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:48.673920 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:48.677942 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:48.683222 (+ 5280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:48.683228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:48.683232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:48.683409 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:48.684125 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:48.684130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:48.684134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:48.684530 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:48.684669 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:48.688185 (+ 3516us) server_negotiation.cc:299] Negotiation successful
0926 21:39:48.688431 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22004,"thread_start_us":22447,"threads_started":1}
I20260926 21:39:48.691548 12932 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39981
I20260926 21:39:48.691908 12932 heartbeater.cc:464] Registering TS with master...
I20260926 21:39:48.692783 12932 heartbeater.cc:511] Master 127.0.97.62:39981 requested a full tablet report, sending...
I20260926 21:39:48.695297 12717 ts_manager.cc:194] Registered new tserver with Master: 270ae1110c174b2399e2e1f79e08050d (127.0.97.1:43911)
I20260926 21:39:48.697345 12717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:39:48.712800 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39981
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:39:49.047720 12940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:49.047981 12940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:49.048141 12940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:49.055203 12940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:49.055305 12940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:49.055524 12940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:39:49.065382 12940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:52289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:49.066766 12940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:49.068243 12940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:49.081419 12950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:49.081399 12953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:49.081429 12951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:49.082702 12940 server_base.cc:1034] running on GCE node
I20260926 21:39:49.083421 12940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:49.084404 12940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:49.085646 12940 hybrid_clock.cc:648] HybridClock initialized: now 1790458789085599 us; error 63 us; skew 500 ppm
I20260926 21:39:49.085989 12940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:49 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:49.089161 12940 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:39:49.090682 12940 webserver.cc:460] Webserver started at http://127.0.97.2:46519/ using document root <none> and password file <none>
I20260926 21:39:49.091369 12940 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:49.091516 12940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:39:49.091840 12940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:39:49 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:49.109303 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "2f91df92189540c6a8fe44ad2dfda0aa"
format_stamp: "Formatted at 2026-09-26 21:39:49 on dist-test-slave-lhhq"
server_key: "6c8e24ee67c7cc695be12dc030934afb"
server_key_iv: "a3e35bf7567778f3a69d4c2e58ef26a8"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:49.110159 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2f91df92189540c6a8fe44ad2dfda0aa"
format_stamp: "Formatted at 2026-09-26 21:39:49 on dist-test-slave-lhhq"
server_key: "6c8e24ee67c7cc695be12dc030934afb"
server_key_iv: "a3e35bf7567778f3a69d4c2e58ef26a8"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:49.115443 12940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:39:49.132228 12960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:49.133859 12940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:39:49.134054 12940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "2f91df92189540c6a8fe44ad2dfda0aa"
format_stamp: "Formatted at 2026-09-26 21:39:49 on dist-test-slave-lhhq"
server_key: "6c8e24ee67c7cc695be12dc030934afb"
server_key_iv: "a3e35bf7567778f3a69d4c2e58ef26a8"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:49.134298 12940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:49.147820 12940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:49.151403 12940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:49.151708 12940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:49.152621 12940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:39:49.154026 12940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:39:49.154182 12940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:49.154347 12940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:39:49.154421 12940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:49.187163 12940 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42261
I20260926 21:39:49.187198 13072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42261 every 8 connection(s)
I20260926 21:39:49.188441 12940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:39:49.190332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 12940
I20260926 21:39:49.190536 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:39:49.191143 388 external_mini_cluster.cc:1428] Setting key 46a40ec44dede64371cb07ea1ab960d1
Sep 26 21:39:49 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:39:49.211627 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:49.191157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56063 (local address 127.0.97.62:39981)
0926 21:39:49.191631 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:49.191638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:49.192652 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 21:39:49.195281 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:49.195285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:49.195345 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:49.195436 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:49.197342 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.198094 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.200532 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.200949 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.201045 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:49.204558 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:49.204567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:49.204575 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:49.204620 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:49.208280 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:49.209051 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:49.209059 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:49.209066 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:49.209265 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:49.209678 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:49.209686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:49.209700 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:49.210085 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:49.210244 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:49.210869 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 21:39:49.211130 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
I20260926 21:39:49.212180 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:49.191691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39981 (local address 127.0.97.2:56063)
0926 21:39:49.192509 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:49.192550 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:49.194988 (+ 2438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:49.195702 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:49.195715 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:49.196324 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:49.197144 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:49.197170 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.198252 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.198260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:49.200386 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:49.200403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.201182 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.201220 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:49.201496 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:49.202166 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:49.202187 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:49.204337 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:49.208411 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:49.208417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:49.208429 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:49.208915 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:49.209372 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:49.209376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:49.209378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:49.209559 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:49.210203 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:49.210208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:49.210492 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 21:39:49.210705 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 21:39:49.211083 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":336,"threads_started":1}
I20260926 21:39:49.213749 13073 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39981
I20260926 21:39:49.214047 13073 heartbeater.cc:464] Registering TS with master...
I20260926 21:39:49.214758 13073 heartbeater.cc:511] Master 127.0.97.62:39981 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:39:49.216336 12717 ts_manager.cc:194] Registered new tserver with Master: 2f91df92189540c6a8fe44ad2dfda0aa (127.0.97.2:42261)
I20260926 21:39:49.217126 12717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:39:49.246063 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39981
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:39:49.570027 13079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:49.570320 13079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:49.570482 13079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:49.577839 13079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:49.577955 13079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:49.578119 13079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:39:49.588238 13079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:52289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:49.589633 13079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:49.590966 13079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:49.603935 13087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:49.604005 13084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:49.603956 13085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:49.605055 13079 server_base.cc:1034] running on GCE node
I20260926 21:39:49.605763 13079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:49.606746 13079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:49.607951 13079 hybrid_clock.cc:648] HybridClock initialized: now 1790458789607908 us; error 40 us; skew 500 ppm
I20260926 21:39:49.608300 13079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:49 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:49.611693 13079 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:39:49.613555 13079 webserver.cc:460] Webserver started at http://127.0.97.3:33849/ using document root <none> and password file <none>
I20260926 21:39:49.614286 13079 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:49.614421 13079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:39:49.614750 13079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 21:39:49 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:49.633157 13079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "be7818137c454a28884c502cd100f75c"
format_stamp: "Formatted at 2026-09-26 21:39:49 on dist-test-slave-lhhq"
server_key: "bdc231e81b4a95f9d40d1a5445a4a912"
server_key_iv: "20d6f88df902fec6c2e8b092ecbfa986"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:49.634420 13079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "be7818137c454a28884c502cd100f75c"
format_stamp: "Formatted at 2026-09-26 21:39:49 on dist-test-slave-lhhq"
server_key: "bdc231e81b4a95f9d40d1a5445a4a912"
server_key_iv: "20d6f88df902fec6c2e8b092ecbfa986"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:49.641870 13079 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 21:39:49.659325 13094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:49.660943 13079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 21:39:49.661149 13079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "be7818137c454a28884c502cd100f75c"
format_stamp: "Formatted at 2026-09-26 21:39:49 on dist-test-slave-lhhq"
server_key: "bdc231e81b4a95f9d40d1a5445a4a912"
server_key_iv: "20d6f88df902fec6c2e8b092ecbfa986"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:49.661475 13079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:49.695681 13079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:49.699543 13079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:49.699849 13079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:49.700795 13079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:39:49.701349 12932 heartbeater.cc:503] Master 127.0.97.62:39981 was elected leader, sending a full tablet report...
I20260926 21:39:49.702473 13079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:39:49.702572 13079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:49.702698 13079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:39:49.702768 13079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:49.742275 13079 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33667
I20260926 21:39:49.742321 13206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33667 every 8 connection(s)
I20260926 21:39:49.743580 13079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:39:49.748805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 13079
I20260926 21:39:49.748970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:39:49.749532 388 external_mini_cluster.cc:1428] Setting key 97e81bc23160bfd3fe27307e6f8e8338
Sep 26 21:39:49 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:39:49.763105 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:49.746103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34149 (local address 127.0.97.62:39981)
0926 21:39:49.746583 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:49.746588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:49.747436 (+ 848us) server_negotiation.cc:407] Connection header received
0926 21:39:49.749509 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:49.749515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:49.749609 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:49.749745 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:49.751268 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.752313 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.754009 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.754437 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.754582 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:49.757877 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:49.757882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:49.757885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:49.757934 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:49.760310 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:49.760993 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:49.760996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:49.760999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:49.761119 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:49.761581 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:49.761585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:49.761587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:49.761878 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:49.761996 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:49.762604 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 21:39:49.762806 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":261,"threads_started":1}
I20260926 21:39:49.764204 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:49.746567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39981 (local address 127.0.97.3:34149)
0926 21:39:49.747272 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:49.747314 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:49.749233 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:49.749991 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:49.749999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:49.750430 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:49.751101 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:49.751118 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.752461 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.752471 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:49.753865 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:49.753876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.754668 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.754676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:49.754920 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:49.755543 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:49.755566 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:49.757639 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:49.760432 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:49.760439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:49.760451 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:49.760878 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:49.761268 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:49.761271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:49.761273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:49.761479 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:49.761979 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:49.761985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:49.762288 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 21:39:49.762507 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 21:39:49.762970 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":281,"threads_started":1}
I20260926 21:39:49.765753 13207 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39981
I20260926 21:39:49.766088 13207 heartbeater.cc:464] Registering TS with master...
I20260926 21:39:49.766841 13207 heartbeater.cc:511] Master 127.0.97.62:39981 requested a full tablet report, sending...
I20260926 21:39:49.768263 12717 ts_manager.cc:194] Registered new tserver with Master: be7818137c454a28884c502cd100f75c (127.0.97.3:33667)
I20260926 21:39:49.769017 12717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34149
I20260926 21:39:49.777506 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:39:49.779204 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 12797
Sep 26 21:39:49 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:39:49.795034 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:49.780257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.0.97.1:43911)
0926 21:39:49.780906 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:49.780913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:49.780947 (+ 34us) server_negotiation.cc:407] Connection header received
0926 21:39:49.781004 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:49.781010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:49.781300 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:49.781439 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:49.782802 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.783819 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.785164 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.785794 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.785893 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:49.790079 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:49.790091 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:49.790105 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:49.790139 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:49.792740 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:49.793411 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:49.793418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:49.793420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:49.793547 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:49.793857 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:49.793860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:49.793862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:49.794169 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:49.794412 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:49.794585 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 21:39:49.794766 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":322,"threads_started":1}
I20260926 21:39:49.796336 12927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:49.880617 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 12797
I20260926 21:39:49.896737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 12940
Sep 26 21:39:49 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:39:49.913633 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:49.897730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.0.97.2:42261)
0926 21:39:49.898234 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:49.898242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:49.898268 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:39:49.898336 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:49.898342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:49.898554 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:49.898752 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:49.899994 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.900978 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.902679 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:49.903249 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:49.903395 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:49.907958 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:49.907971 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:49.907987 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:49.908026 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:49.910873 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:49.911599 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:49.911607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:49.911609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:49.911733 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:49.912089 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:49.912092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:49.912094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:49.912423 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:49.912645 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:49.913001 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 21:39:49.913253 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":242,"threads_started":1}
I20260926 21:39:49.914927 13068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:49.996959 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 12940
I20260926 21:39:50.012504 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 13079
Sep 26 21:39:50 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:39:50.030257 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:50.013545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.0.97.3:33667)
0926 21:39:50.014217 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:50.014224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:50.014262 (+ 38us) server_negotiation.cc:407] Connection header received
0926 21:39:50.014320 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:50.014327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:50.014540 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:50.014717 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:50.016131 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:50.017159 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:50.018567 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:50.019143 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:50.019304 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:50.023880 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:50.023892 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:50.023908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:50.023945 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:50.027178 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:50.027863 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:50.027870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:50.027873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:50.028000 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:50.028322 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:50.028325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:50.028327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:50.028787 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:50.029124 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:50.029735 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 21:39:50.029956 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":387,"threads_started":1}
I20260926 21:39:50.031558 13202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:50.114684 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 13079
I20260926 21:39:50.130775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 12685
I20260926 21:39:50.132059 12747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:50.238607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 12685
I20260926 21:39:50.260902 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39981
--webserver_interface=127.0.97.62
--webserver_port=39817
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39981
--ranger_config_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52289/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52289/kms with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:39:50.591607 13219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:50.591907 13219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:50.591993 13219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:50.599309 13219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:39:50.599422 13219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:50.599469 13219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:50.599519 13219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:39:50.599565 13219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:39:50.609834 13219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52289/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39981
--ranger_config_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:50.611093 13219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:50.612375 13219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:50.624480 13224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:50.624435 13227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:50.624518 13225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:50.625890 13219 server_base.cc:1034] running on GCE node
I20260926 21:39:50.626686 13219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:50.628193 13219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:50.629448 13219 hybrid_clock.cc:648] HybridClock initialized: now 1790458790629392 us; error 72 us; skew 500 ppm
I20260926 21:39:50.629797 13219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:50 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:50.632934 13219 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:39:50.634461 13219 webserver.cc:460] Webserver started at http://127.0.97.62:39817/ using document root <none> and password file <none>
I20260926 21:39:50.635099 13219 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:50.635211 13219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 21:39:50 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:50.698307 13219 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:39:50.701306 13235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:50.702732 13219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:39:50.702930 13219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "3ac1447611fb47f59194e0c32a4523e4"
format_stamp: "Formatted at 2026-09-26 21:39:46 on dist-test-slave-lhhq"
server_key: "aba2683a95d08f65fcb1571f9403967b"
server_key_iv: "66c49c4f1ceaae37897ca25ce804eb66"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:50.703964 13219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:50.726122 13219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:50.730216 13219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:50.730522 13219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:50.750067 13219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39981
I20260926 21:39:50.750092 13286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39981 every 8 connection(s)
I20260926 21:39:50.751631 13219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:39:50.754719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 13219
I20260926 21:39:50.755599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43911
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39981
--webserver_port=38057
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52289/kms with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:39:50.758716 13287 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:39:50.762122 13287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Bootstrap starting.
I20260926 21:39:50.781452 13287 log.cc:824] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Log is configured to *not* fsync() on all Append() calls
I20260926 21:39:50.790409 13287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:39:50.790973 13287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Bootstrap complete.
I20260926 21:39:50.800743 13287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } }
I20260926 21:39:50.801795 13287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac1447611fb47f59194e0c32a4523e4, State: Initialized, Role: FOLLOWER
I20260926 21:39:50.802441 13287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } }
I20260926 21:39:50.802693 13287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:39:50.802830 13287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:39:50.803005 13287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:39:50.807195 13287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } }
I20260926 21:39:50.808043 13287 leader_election.cc:302] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ac1447611fb47f59194e0c32a4523e4; no voters:
I20260926 21:39:50.809105 13287 leader_election.cc:288] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 21:39:50.809290 13291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:39:50.813289 13287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:39:50.813488 13291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [term 2 LEADER]: Becoming Leader. State: Replica: 3ac1447611fb47f59194e0c32a4523e4, State: Running, Role: LEADER
I20260926 21:39:50.814330 13291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } }
I20260926 21:39:50.817420 13292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3ac1447611fb47f59194e0c32a4523e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } } }
I20260926 21:39:50.817796 13292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: This master's current role is: LEADER
I20260926 21:39:50.817112 13293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ac1447611fb47f59194e0c32a4523e4. Latest consensus state: current_term: 2 leader_uuid: "3ac1447611fb47f59194e0c32a4523e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1447611fb47f59194e0c32a4523e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39981 } } }
I20260926 21:39:50.818409 13293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4 [sys.catalog]: This master's current role is: LEADER
I20260926 21:39:50.825544 13301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:39:50.826045 13287 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 21:39:50.840684 13301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:39:50.842960 13301 catalog_manager.cc:1259] Loaded cluster ID: 297c425b54a94fe58b76848ccd050be3
I20260926 21:39:50.843137 13301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:39:50.848201 13301 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:39:50.850015 13301 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Loaded TSK: 0
I20260926 21:39:50.851897 13301 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 21:39:51.232275 13289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:51.232703 13289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:51.232940 13289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:51.244978 13289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:51.245203 13289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:51.245432 13289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:39:51.261541 13289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38057
--tserver_master_addrs=127.0.97.62:39981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:51.263540 13289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:51.265468 13289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:51.289811 13324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:51.289987 13325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:51.291337 13327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:51.292073 13289 server_base.cc:1034] running on GCE node
I20260926 21:39:51.292866 13289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:51.294181 13289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:51.295445 13289 hybrid_clock.cc:648] HybridClock initialized: now 1790458791295363 us; error 62 us; skew 500 ppm
I20260926 21:39:51.295873 13289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:51 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:51.307190 13289 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:39:51.309404 13289 webserver.cc:460] Webserver started at http://127.0.97.1:38057/ using document root <none> and password file <none>
I20260926 21:39:51.310434 13289 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:51.310611 13289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 21:39:51 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:51.338410 13289 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.001s
I20260926 21:39:51.342903 13335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:51.344848 13289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 21:39:51.345059 13289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "270ae1110c174b2399e2e1f79e08050d"
format_stamp: "Formatted at 2026-09-26 21:39:47 on dist-test-slave-lhhq"
server_key: "b335689bc166ee0dc0f8fa0d4fbb8e5b"
server_key_iv: "4382eb595e296701ac75bd3e3049f283"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:51.346541 13289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:51.376013 13289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:51.381330 13289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:51.381652 13289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:51.382893 13289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:39:51.384744 13289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:39:51.384886 13289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:51.385010 13289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:39:51.385092 13289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:51.469493 13289 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43911
I20260926 21:39:51.469543 13448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43911 every 8 connection(s)
I20260926 21:39:51.471277 13289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:39:51.474645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 13289
I20260926 21:39:51.475445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42261
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39981
--webserver_port=46519
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52289/kms with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:39:51 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:39:51.517433 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:51.475085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39981 (local address 127.0.97.1:36425)
0926 21:39:51.481905 (+ 6820us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:51.481977 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:51.486153 (+ 4176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:51.488792 (+ 2639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:51.488801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:51.489969 (+ 1168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:51.490563 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:51.490581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:51.492817 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:51.492823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:51.494330 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:51.494351 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:51.495661 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:51.495670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:51.495906 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:51.496646 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:51.496671 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:51.498990 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:51.503561 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:51.503571 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:51.503589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:51.504282 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:51.505072 (+ 790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:51.505080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:51.505086 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:51.505463 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:51.506858 (+ 1395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:51.506867 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:51.507363 (+ 496us) client_negotiation.cc:769] Sending connection context
0926 21:39:51.515210 (+ 7847us) client_negotiation.cc:240] Negotiation successful
0926 21:39:51.515894 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5275,"thread_start_us":4997,"threads_started":1}
I20260926 21:39:51.520153 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:51.476297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36425 (local address 127.0.97.62:39981)
0926 21:39:51.479974 (+ 3677us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:51.480004 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:51.482205 (+ 2201us) server_negotiation.cc:407] Connection header received
0926 21:39:51.487733 (+ 5528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:51.487748 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:51.488146 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:51.488574 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:51.490806 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:51.492626 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:51.494509 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:51.495254 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:51.495467 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:51.499288 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:51.499297 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:51.499314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:51.499354 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:51.503356 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:51.504508 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:51.504522 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:51.504534 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:51.504890 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:51.505675 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:51.505683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:51.505689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:51.506497 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:51.506763 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:51.518435 (+ 11672us) server_negotiation.cc:299] Negotiation successful
0926 21:39:51.518901 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3315,"thread_start_us":3096,"threads_started":1}
I20260926 21:39:51.523478 13449 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39981
I20260926 21:39:51.523836 13449 heartbeater.cc:464] Registering TS with master...
I20260926 21:39:51.524777 13449 heartbeater.cc:511] Master 127.0.97.62:39981 requested a full tablet report, sending...
W20260926 21:39:51.528169 13449 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39981 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 21:39:51.528404 13449 heartbeater.cc:464] Registering TS with master...
I20260926 21:39:51.528713 13449 heartbeater.cc:511] Master 127.0.97.62:39981 requested a full tablet report, sending...
I20260926 21:39:51.530001 13449 heartbeater.cc:464] Registering TS with master...
I20260926 21:39:51.530354 13449 heartbeater.cc:511] Master 127.0.97.62:39981 requested a full tablet report, sending...
W20260926 21:39:51.531461 13449 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:39:51.898298 13453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:51.898628 13453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:51.898792 13453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:51.907308 13453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:51.907465 13453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:51.907655 13453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:39:51.918111 13453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46519
--tserver_master_addrs=127.0.97.62:39981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:51.919565 13453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:51.920974 13453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:51.938118 13468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:51.941460 13470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:51.941962 13467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:51.942140 13453 server_base.cc:1034] running on GCE node
I20260926 21:39:51.942854 13453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:51.965552 13453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:51.966967 13453 hybrid_clock.cc:648] HybridClock initialized: now 1790458791966858 us; error 113 us; skew 500 ppm
I20260926 21:39:51.967449 13453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:51 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:51.971913 13453 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:39:51.993772 13453 webserver.cc:460] Webserver started at http://127.0.97.2:46519/ using document root <none> and password file <none>
I20260926 21:39:51.994872 13453 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:51.995112 13453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 21:39:52 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:52.022049 13453 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:39:52.025789 13480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:52.027622 13453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 21:39:52.027812 13453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "2f91df92189540c6a8fe44ad2dfda0aa"
format_stamp: "Formatted at 2026-09-26 21:39:49 on dist-test-slave-lhhq"
server_key: "6c8e24ee67c7cc695be12dc030934afb"
server_key_iv: "a3e35bf7567778f3a69d4c2e58ef26a8"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:52.029204 13453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:52.045073 13453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:52.049263 13453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:52.049554 13453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:52.050551 13453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:39:52.052096 13453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
Sep 26 21:39:52 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790458792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:52.052286 13453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:52.052435 13453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:39:52.052524 13453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:52.113118 13453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42261
I20260926 21:39:52.113165 13592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42261 every 8 connection(s)
I20260926 21:39:52.114796 13453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:39:52.117843 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 13453
I20260926 21:39:52.118609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33667
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39981
--webserver_port=33849
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52289/kms with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:39:52 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:39:52.149453 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:52.118427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39981 (local address 127.0.97.2:42957)
0926 21:39:52.119230 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:52.119277 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:52.120954 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:52.130159 (+ 9205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:52.130176 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:52.130884 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:52.131753 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:52.131781 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:52.133564 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:52.133570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:52.135597 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:52.135610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:52.136464 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:52.136472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:52.136708 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:52.137502 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:52.137534 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:52.140207 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:52.144504 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:52.144514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:52.144533 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:52.145156 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:52.145676 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:52.145681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:52.145685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:52.145948 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:52.146629 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:52.146638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:52.147081 (+ 443us) client_negotiation.cc:769] Sending connection context
0926 21:39:52.147401 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 21:39:52.147973 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":338,"threads_started":1}
I20260926 21:39:52.150902 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:52.125422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42957 (local address 127.0.97.62:39981)
0926 21:39:52.129413 (+ 3991us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:52.129420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:52.129446 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:39:52.129531 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:52.129537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:52.129628 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:52.131966 (+ 2338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:52.132147 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:52.133351 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:52.135763 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:52.136207 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:52.136355 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:52.140495 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:52.140502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:52.140506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:52.140549 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:52.144326 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:52.145377 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:52.145383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:52.145388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:52.145544 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:52.146099 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:52.146104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:52.146108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:52.146470 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:52.150178 (+ 3708us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:52.150325 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 21:39:52.150551 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3852,"thread_start_us":3682,"threads_started":1}
I20260926 21:39:52.152518 13593 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39981
I20260926 21:39:52.152842 13593 heartbeater.cc:464] Registering TS with master...
I20260926 21:39:52.153693 13593 heartbeater.cc:511] Master 127.0.97.62:39981 requested a full tablet report, sending...
I20260926 21:39:52.155741 13252 ts_manager.cc:194] Registered new tserver with Master: 2f91df92189540c6a8fe44ad2dfda0aa (127.0.97.2:42261)
I20260926 21:39:52.157980 13252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42957
Sep 26 21:39:52 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:52.531618 13449 heartbeater.cc:464] Registering TS with master...
I20260926 21:39:52.532325 13449 heartbeater.cc:511] Master 127.0.97.62:39981 requested a full tablet report, sending...
I20260926 21:39:52.533824 13252 ts_manager.cc:194] Registered new tserver with Master: 270ae1110c174b2399e2e1f79e08050d (127.0.97.1:43911)
I20260926 21:39:52.535373 13252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36425
W20260926 21:39:52.570117 13597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:52.570614 13597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:52.570809 13597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:52.582230 13597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:52.582470 13597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:52.582692 13597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:39:52.598280 13597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33849
--tserver_master_addrs=127.0.97.62:39981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:52.600203 13597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:52.601970 13597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:52.635421 13604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:52.641718 13607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:52.645071 13597 server_base.cc:1034] running on GCE node
W20260926 21:39:52.645099 13603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:52.646169 13597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:52.652729 13597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:52.654012 13597 hybrid_clock.cc:648] HybridClock initialized: now 1790458792653934 us; error 83 us; skew 500 ppm
I20260926 21:39:52.654450 13597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:52 dist-test-slave-lhhq krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:52.658761 13597 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:39:52.676371 13597 webserver.cc:460] Webserver started at http://127.0.97.3:33849/ using document root <none> and password file <none>
I20260926 21:39:52.677518 13597 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:52.677762 13597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 21:39:52 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:52.711701 13597 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:39:52.715518 13614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:52.717423 13597 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 21:39:52.717617 13597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "be7818137c454a28884c502cd100f75c"
format_stamp: "Formatted at 2026-09-26 21:39:49 on dist-test-slave-lhhq"
server_key: "bdc231e81b4a95f9d40d1a5445a4a912"
server_key_iv: "20d6f88df902fec6c2e8b092ecbfa986"
server_key_version: "kuduclusterkey@0"
I20260926 21:39:52.719050 13597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:52.754627 13597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:52.759959 13597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:52.760273 13597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:52.761399 13597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:39:52.762949 13597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:39:52.763072 13597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:52.763216 13597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:39:52.763306 13597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 21:39:52 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 21:39:53.162662 13593 heartbeater.cc:503] Master 127.0.97.62:39981 was elected leader, sending a full tablet report...
I20260926 21:39:53.175067 13727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33667 every 8 connection(s)
I20260926 21:39:53.175444 13597 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33667
I20260926 21:39:53.177510 13597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:39:53.179176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 13597
Sep 26 21:39:53 dist-test-slave-lhhq krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:39:53.251974 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.180316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.0.97.62:39981)
0926 21:39:53.205709 (+ 25393us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.205718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.205751 (+ 33us) server_negotiation.cc:407] Connection header received
0926 21:39:53.205841 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.205849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.205956 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.206129 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:53.219939 (+ 13810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.221493 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.229685 (+ 8192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.230400 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.230551 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.241207 (+ 10656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:53.241216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:53.241223 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:53.241275 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:53.245251 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.247106 (+ 1855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.247111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.247115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.247288 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.248541 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.248548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.248553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.248976 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:53.249313 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.251294 (+ 1981us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.251519 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25188,"spinlock_wait_cycles":1280,"thread_start_us":25951,"threads_started":1}
I20260926 21:39:53.256922 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.199519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53169 (local address 127.0.97.62:39981)
0926 21:39:53.227433 (+ 27914us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.227441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.233261 (+ 5820us) server_negotiation.cc:407] Connection header received
0926 21:39:53.235008 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.235014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.235107 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.235227 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:53.237817 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.239057 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.241674 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.242396 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.242503 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.247212 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:53.247219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:53.247223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:53.247278 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:53.250687 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.252807 (+ 2120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.252812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.252815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.252959 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.254285 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.254290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.254293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.254629 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:53.254776 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.256314 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.256541 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27749,"spinlock_wait_cycles":4608,"thread_start_us":27935,"threads_started":1}
I20260926 21:39:53.259550 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.198884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39981 (local address 127.0.97.3:53169)
0926 21:39:53.230373 (+ 31489us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:53.230444 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:53.232881 (+ 2437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:53.235883 (+ 3002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:53.235896 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:53.236686 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:53.237585 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.237613 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.239288 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.239294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.241484 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.241505 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.242974 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.242982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.243321 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.243993 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:53.244025 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:53.246905 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:53.251880 (+ 4975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:53.251891 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:53.251911 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:53.252569 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:53.253106 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:53.253110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:53.253114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:53.254149 (+ 1035us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:53.254881 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:53.254891 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:53.255784 (+ 893us) client_negotiation.cc:769] Sending connection context
0926 21:39:53.257495 (+ 1711us) client_negotiation.cc:240] Negotiation successful
0926 21:39:53.258070 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31063,"thread_start_us":35017,"threads_started":1}
I20260926 21:39:53.261116 13728 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39981
I20260926 21:39:53.261512 13728 heartbeater.cc:464] Registering TS with master...
I20260926 21:39:53.262449 13728 heartbeater.cc:511] Master 127.0.97.62:39981 requested a full tablet report, sending...
I20260926 21:39:53.264222 13252 ts_manager.cc:194] Registered new tserver with Master: be7818137c454a28884c502cd100f75c (127.0.97.3:33667)
I20260926 21:39:53.265425 13252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53169
I20260926 21:39:53.267737 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:39:53.290675 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.271394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.0.97.62:39981)
0926 21:39:53.271583 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.271589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.273801 (+ 2212us) server_negotiation.cc:407] Connection header received
0926 21:39:53.273973 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.273979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.274078 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.274208 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:53.276187 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.277347 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.279415 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.280037 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.280195 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.283631 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:53.283638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:53.283642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:53.283685 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:53.287596 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.288424 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.288430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.288433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.288592 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.289043 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.289049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.289052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.289450 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:53.289692 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.290090 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.290333 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:39:53.297447 13252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:39:53.309268 13252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:39:53.369885 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.329422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.0.97.2:42261)
0926 21:39:53.357395 (+ 27973us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.357408 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.360359 (+ 2951us) server_negotiation.cc:407] Connection header received
0926 21:39:53.360476 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.360484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.360771 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.360974 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:39:53.362230 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.363777 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.366673 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.368308 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.368441 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.368682 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.369307 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.369541 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27728,"thread_start_us":39403,"threads_started":1}
I20260926 21:39:53.372319 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.329070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42261 (local address 127.0.0.1:38824)
0926 21:39:53.360059 (+ 30989us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:53.360080 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:53.360223 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:53.361539 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:53.361547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:53.361579 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:39:53.362013 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.362025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.363945 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.363951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.366450 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.366468 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.368628 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.368637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.368925 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.368986 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 21:39:53.371688 (+ 2702us) client_negotiation.cc:240] Negotiation successful
0926 21:39:53.371912 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30867,"thread_start_us":37281,"threads_started":1}
I20260926 21:39:53.376861 13528 tablet_service.cc:1467] Processing CreateTablet for tablet f7eae74a90104afe843be5bb3c187ac1 (DEFAULT_TABLE table=test-table [id=3c64dbca545d41d4aa6dca1ae903c38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:39:53.378610 13528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7eae74a90104afe843be5bb3c187ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:39:53.381592 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.329961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43911 (local address 127.0.0.1:35818)
0926 21:39:53.366704 (+ 36743us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:53.366726 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:53.366845 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:53.369709 (+ 2864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:53.369713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:53.369741 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:39:53.370050 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.370060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.371926 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.371932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.373908 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.373924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.380779 (+ 6855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.380785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.380960 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.380995 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:39:53.381067 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 21:39:53.381236 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36579,"thread_start_us":28261,"threads_started":1}
I20260926 21:39:53.382871 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.330034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.0.97.1:43911)
0926 21:39:53.356588 (+ 26554us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.356602 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.367018 (+ 10416us) server_negotiation.cc:407] Connection header received
0926 21:39:53.368927 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.368936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.369243 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.369447 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:39:53.370260 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.371775 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.374111 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.375560 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.382013 (+ 6453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.382275 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.382352 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.382547 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26284,"thread_start_us":26589,"threads_started":1}
I20260926 21:39:53.386910 13384 tablet_service.cc:1467] Processing CreateTablet for tablet f7eae74a90104afe843be5bb3c187ac1 (DEFAULT_TABLE table=test-table [id=3c64dbca545d41d4aa6dca1ae903c38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:39:53.388470 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7eae74a90104afe843be5bb3c187ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:39:53.398316 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.328649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33667 (local address 127.0.0.1:52676)
0926 21:39:53.358647 (+ 29998us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:53.358714 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:53.358972 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:53.367107 (+ 8135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:53.367113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:53.367147 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:39:53.367485 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.367496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.376471 (+ 8975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.376477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.378470 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.378488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.397425 (+ 18937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.397433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.397644 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.397696 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 21:39:53.397806 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 21:39:53.397962 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29717,"thread_start_us":32567,"threads_started":1}
I20260926 21:39:53.399384 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.329475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.0.97.3:33667)
0926 21:39:53.357893 (+ 28418us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.357904 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.365273 (+ 7369us) server_negotiation.cc:407] Connection header received
0926 21:39:53.365385 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.365393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.365721 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.365943 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:39:53.367744 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.376288 (+ 8544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.378715 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.380169 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.380305 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.380546 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.398852 (+ 18306us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.399055 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28241,"thread_start_us":28094,"threads_started":1}
I20260926 21:39:53.403493 13663 tablet_service.cc:1467] Processing CreateTablet for tablet f7eae74a90104afe843be5bb3c187ac1 (DEFAULT_TABLE table=test-table [id=3c64dbca545d41d4aa6dca1ae903c38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:39:53.405021 13663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7eae74a90104afe843be5bb3c187ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:39:53.425315 13756 tablet_bootstrap.cc:492] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: Bootstrap starting.
I20260926 21:39:53.429276 13756 tablet_bootstrap.cc:654] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: Neither blocks nor log segments found. Creating new log.
I20260926 21:39:53.430696 13756 log.cc:824] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: Log is configured to *not* fsync() on all Append() calls
I20260926 21:39:53.434370 13757 tablet_bootstrap.cc:492] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: Bootstrap starting.
I20260926 21:39:53.434826 13759 tablet_bootstrap.cc:492] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: Bootstrap starting.
I20260926 21:39:53.439097 13757 tablet_bootstrap.cc:654] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: Neither blocks nor log segments found. Creating new log.
I20260926 21:39:53.440550 13757 log.cc:824] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: Log is configured to *not* fsync() on all Append() calls
I20260926 21:39:53.440917 13759 tablet_bootstrap.cc:654] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: Neither blocks nor log segments found. Creating new log.
I20260926 21:39:53.445832 13756 tablet_bootstrap.cc:492] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: No bootstrap required, opened a new log
I20260926 21:39:53.446230 13756 ts_tablet_manager.cc:1397] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.006s
I20260926 21:39:53.453318 13759 log.cc:824] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: Log is configured to *not* fsync() on all Append() calls
I20260926 21:39:53.460346 13757 tablet_bootstrap.cc:492] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: No bootstrap required, opened a new log
I20260926 21:39:53.460379 13759 tablet_bootstrap.cc:492] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: No bootstrap required, opened a new log
I20260926 21:39:53.460683 13759 ts_tablet_manager.cc:1397] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.003s
I20260926 21:39:53.460765 13757 ts_tablet_manager.cc:1397] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.009s
I20260926 21:39:53.471397 13757 raft_consensus.cc:348] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } }
I20260926 21:39:53.473662 13759 raft_consensus.cc:348] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } }
I20260926 21:39:53.473960 13756 raft_consensus.cc:348] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } }
I20260926 21:39:53.474272 13759 raft_consensus.cc:374] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:39:53.474411 13759 raft_consensus.cc:729] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7818137c454a28884c502cd100f75c, State: Initialized, Role: FOLLOWER
I20260926 21:39:53.474439 13756 raft_consensus.cc:374] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:39:53.474524 13756 raft_consensus.cc:729] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f91df92189540c6a8fe44ad2dfda0aa, State: Initialized, Role: FOLLOWER
I20260926 21:39:53.475332 13759 consensus_queue.cc:260] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } }
I20260926 21:39:53.475332 13756 consensus_queue.cc:260] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } }
I20260926 21:39:53.478832 13757 raft_consensus.cc:374] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:39:53.480878 13756 ts_tablet_manager.cc:1428] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: Time spent starting tablet: real 0.034s user 0.011s sys 0.004s
I20260926 21:39:53.481220 13757 raft_consensus.cc:729] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270ae1110c174b2399e2e1f79e08050d, State: Initialized, Role: FOLLOWER
I20260926 21:39:53.483867 13757 consensus_queue.cc:260] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } }
I20260926 21:39:53.489806 13759 ts_tablet_manager.cc:1428] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: Time spent starting tablet: real 0.029s user 0.013s sys 0.006s
I20260926 21:39:53.490300 13728 heartbeater.cc:503] Master 127.0.97.62:39981 was elected leader, sending a full tablet report...
I20260926 21:39:53.496547 13449 heartbeater.cc:503] Master 127.0.97.62:39981 was elected leader, sending a full tablet report...
I20260926 21:39:53.497381 13757 ts_tablet_manager.cc:1428] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: Time spent starting tablet: real 0.036s user 0.015s sys 0.010s
I20260926 21:39:53.596745 13762 raft_consensus.cc:482] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:39:53.597100 13762 raft_consensus.cc:504] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } }
I20260926 21:39:53.599331 13762 leader_election.cc:288] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be7818137c454a28884c502cd100f75c (127.0.97.3:33667), 270ae1110c174b2399e2e1f79e08050d (127.0.97.1:43911)
W20260926 21:39:53.617370 13594 tablet.cc:2369] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:39:53.620831 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.600040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57027 (local address 127.0.97.3:33667)
0926 21:39:53.600239 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.600245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.611704 (+ 11459us) server_negotiation.cc:407] Connection header received
0926 21:39:53.612233 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.612238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.612316 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.612376 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:39:53.613274 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.614387 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.618445 (+ 4058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.619631 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.619731 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.619838 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.620416 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.620576 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:39:53.620913 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.600440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56339 (local address 127.0.97.1:43911)
0926 21:39:53.600636 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.600642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.613557 (+ 12915us) server_negotiation.cc:407] Connection header received
0926 21:39:53.613826 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.613832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.613922 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.613986 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:39:53.614789 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.615872 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.618511 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.619716 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.619824 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.619926 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.620527 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.620668 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:39:53.621711 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.599930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43911 (local address 127.0.97.2:56339)
0926 21:39:53.613437 (+ 13507us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:53.613458 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:53.613660 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:53.614178 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:53.614182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:53.614207 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:39:53.614583 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.614594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.616063 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.616068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.618327 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.618342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.620141 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.620148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.620332 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.620365 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:39:53.621154 (+ 789us) client_negotiation.cc:240] Negotiation successful
0926 21:39:53.621345 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13141,"thread_start_us":13277,"threads_started":1}
I20260926 21:39:53.622046 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.599648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33667 (local address 127.0.97.2:57027)
0926 21:39:53.611557 (+ 11909us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:53.611585 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:53.612040 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:53.612580 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:53.612585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:53.612612 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:39:53.613057 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.613074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.614647 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.614652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.616935 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:53.616950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.620034 (+ 3084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.620040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:53.620232 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.620265 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:39:53.621384 (+ 1119us) client_negotiation.cc:240] Negotiation successful
0926 21:39:53.621535 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11744,"thread_start_us":12076,"threads_started":1}
I20260926 21:39:53.622824 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7eae74a90104afe843be5bb3c187ac1" candidate_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270ae1110c174b2399e2e1f79e08050d" is_pre_election: true
I20260926 21:39:53.623155 13683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7eae74a90104afe843be5bb3c187ac1" candidate_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7818137c454a28884c502cd100f75c" is_pre_election: true
I20260926 21:39:53.623478 13404 raft_consensus.cc:2454] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f91df92189540c6a8fe44ad2dfda0aa in term 0.
I20260926 21:39:53.623674 13683 raft_consensus.cc:2454] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f91df92189540c6a8fe44ad2dfda0aa in term 0.
I20260926 21:39:53.624318 13483 leader_election.cc:302] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 270ae1110c174b2399e2e1f79e08050d, 2f91df92189540c6a8fe44ad2dfda0aa; no voters:
I20260926 21:39:53.625037 13762 raft_consensus.cc:2781] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:39:53.625207 13762 raft_consensus.cc:482] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:39:53.625322 13762 raft_consensus.cc:3037] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:39:53.627516 13762 raft_consensus.cc:504] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } }
I20260926 21:39:53.628175 13762 leader_election.cc:288] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [CANDIDATE]: Term 1 election: Requested vote from peers be7818137c454a28884c502cd100f75c (127.0.97.3:33667), 270ae1110c174b2399e2e1f79e08050d (127.0.97.1:43911)
I20260926 21:39:53.628590 13683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7eae74a90104afe843be5bb3c187ac1" candidate_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7818137c454a28884c502cd100f75c"
I20260926 21:39:53.628765 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7eae74a90104afe843be5bb3c187ac1" candidate_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270ae1110c174b2399e2e1f79e08050d"
I20260926 21:39:53.628819 13683 raft_consensus.cc:3037] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:39:53.628964 13404 raft_consensus.cc:3037] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:39:53.632113 13683 raft_consensus.cc:2454] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f91df92189540c6a8fe44ad2dfda0aa in term 1.
I20260926 21:39:53.632113 13404 raft_consensus.cc:2454] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f91df92189540c6a8fe44ad2dfda0aa in term 1.
I20260926 21:39:53.632584 13483 leader_election.cc:302] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 270ae1110c174b2399e2e1f79e08050d, 2f91df92189540c6a8fe44ad2dfda0aa; no voters:
I20260926 21:39:53.633272 13762 raft_consensus.cc:2781] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:39:53.633850 13762 raft_consensus.cc:686] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 1 LEADER]: Becoming Leader. State: Replica: 2f91df92189540c6a8fe44ad2dfda0aa, State: Running, Role: LEADER
I20260926 21:39:53.634663 13762 consensus_queue.cc:237] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } }
I20260926 21:39:53.639690 13251 catalog_manager.cc:5495] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f91df92189540c6a8fe44ad2dfda0aa (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f91df92189540c6a8fe44ad2dfda0aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42261 } health_report { overall_health: HEALTHY } } }
I20260926 21:39:53.672658 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.666662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.0.97.2:42261)
0926 21:39:53.666886 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.666892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.667046 (+ 154us) server_negotiation.cc:407] Connection header received
0926 21:39:53.667304 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.667308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.667447 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.667625 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:39:53.668060 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.668952 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.670392 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.670874 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.671015 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.671552 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:39:53.672073 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:39:53.672245 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.672298 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.672377 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:39:53.681334 13404 raft_consensus.cc:1261] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [term 1 FOLLOWER]: Refusing update from remote peer 2f91df92189540c6a8fe44ad2dfda0aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:39:53.681334 13683 raft_consensus.cc:1261] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [term 1 FOLLOWER]: Refusing update from remote peer 2f91df92189540c6a8fe44ad2dfda0aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:39:53.682428 13767 consensus_queue.cc:1035] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "be7818137c454a28884c502cd100f75c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:39:53.682791 13762 consensus_queue.cc:1035] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "270ae1110c174b2399e2e1f79e08050d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:39:53.698284 13770 mvcc.cc:205] Tried to move back new op lower bound from 7333719218908401664 to 7333719218732142592. Current Snapshot: MvccSnapshot[applied={T|T < 7333719218908401664}]
I20260926 21:39:53.729677 13251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:39:53.730007 13251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:39:53.734301 13251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ac1447611fb47f59194e0c32a4523e4: Sending DeleteTablet for 3 replicas of tablet f7eae74a90104afe843be5bb3c187ac1
I20260926 21:39:53.735553 13663 tablet_service.cc:1514] Processing DeleteTablet for tablet f7eae74a90104afe843be5bb3c187ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:39:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52676
I20260926 21:39:53.735810 13528 tablet_service.cc:1514] Processing DeleteTablet for tablet f7eae74a90104afe843be5bb3c187ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:39:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38824
I20260926 21:39:53.736347 13384 tablet_service.cc:1514] Processing DeleteTablet for tablet f7eae74a90104afe843be5bb3c187ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:39:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35818
I20260926 21:39:53.744519 13782 tablet_replica.cc:330] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: stopping tablet replica
I20260926 21:39:53.744982 13782 raft_consensus.cc:2229] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:39:53.745584 13782 raft_consensus.cc:2258] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:39:53.746824 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 13289
I20260926 21:39:53.751677 13782 ts_tablet_manager.cc:1905] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:39:53.759140 13782 ts_tablet_manager.cc:1918] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:39:53.759847 13781 tablet_replica.cc:330] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: stopping tablet replica
I20260926 21:39:53.760457 13781 raft_consensus.cc:2229] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:39:53.761349 13781 raft_consensus.cc:2258] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:39:53.769656 13782 log.cc:1196] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/f7eae74a90104afe843be5bb3c187ac1
I20260926 21:39:53.770949 13782 ts_tablet_manager.cc:1939] T f7eae74a90104afe843be5bb3c187ac1 P 270ae1110c174b2399e2e1f79e08050d: Deleting consensus metadata
I20260926 21:39:53.773399 13238 catalog_manager.cc:4841] TS 270ae1110c174b2399e2e1f79e08050d (127.0.97.1:43911): tablet f7eae74a90104afe843be5bb3c187ac1 (table test-table [id=3c64dbca545d41d4aa6dca1ae903c38c]) successfully deleted
I20260926 21:39:53.773646 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.749468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.0.97.1:43911)
0926 21:39:53.749648 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.749654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.749682 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:39:53.749745 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.749751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.749834 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.749933 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:53.753148 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.754278 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.756098 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.756784 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.756914 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.763037 (+ 6123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:53.763053 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:53.763071 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:53.763111 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:53.767069 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.768050 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.768061 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.768065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.768219 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.769620 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.769626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.769632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.770174 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:53.770512 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.772619 (+ 2107us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.772858 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:39:53.774367 13444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:53.776044 13781 ts_tablet_manager.cc:1905] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:39:53.779189 13780 tablet_replica.cc:330] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: stopping tablet replica
I20260926 21:39:53.779873 13780 raft_consensus.cc:2229] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:39:53.780427 13780 raft_consensus.cc:2258] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:39:53.784695 13780 ts_tablet_manager.cc:1905] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:39:53.785030 13781 ts_tablet_manager.cc:1918] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:39:53.785245 13781 log.cc:1196] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/f7eae74a90104afe843be5bb3c187ac1
I20260926 21:39:53.792997 13780 ts_tablet_manager.cc:1918] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:39:53.793322 13780 log.cc:1196] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/f7eae74a90104afe843be5bb3c187ac1
I20260926 21:39:53.793947 13780 ts_tablet_manager.cc:1939] T f7eae74a90104afe843be5bb3c187ac1 P be7818137c454a28884c502cd100f75c: Deleting consensus metadata
I20260926 21:39:53.796188 13236 catalog_manager.cc:4841] TS be7818137c454a28884c502cd100f75c (127.0.97.3:33667): tablet f7eae74a90104afe843be5bb3c187ac1 (table test-table [id=3c64dbca545d41d4aa6dca1ae903c38c]) successfully deleted
I20260926 21:39:53.801606 13781 ts_tablet_manager.cc:1939] T f7eae74a90104afe843be5bb3c187ac1 P 2f91df92189540c6a8fe44ad2dfda0aa: Deleting consensus metadata
I20260926 21:39:53.803503 13239 catalog_manager.cc:4841] TS 2f91df92189540c6a8fe44ad2dfda0aa (127.0.97.2:42261): tablet f7eae74a90104afe843be5bb3c187ac1 (table test-table [id=3c64dbca545d41d4aa6dca1ae903c38c]) successfully deleted
I20260926 21:39:53.936597 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 13289
I20260926 21:39:53.960551 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 13453
I20260926 21:39:53.984447 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:53.961420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.0.97.2:42261)
0926 21:39:53.961647 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:53.961654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:53.961748 (+ 94us) server_negotiation.cc:407] Connection header received
0926 21:39:53.961931 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:53.961937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:53.962035 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:53.962144 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:53.963847 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.965097 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.967160 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:53.967891 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:53.968035 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:53.976683 (+ 8648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:53.976704 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:53.976725 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:53.976782 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:53.981073 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.982038 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.982048 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.982052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.982240 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:53.982667 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:53.982672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:53.982676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:53.983150 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:53.983478 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:53.983783 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 21:39:53.984047 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 21:39:53.985154 13588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:54.097479 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 13453
I20260926 21:39:54.118427 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 13597
I20260926 21:39:54.133229 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:54.119343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.0.97.3:33667)
0926 21:39:54.119544 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:54.119551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:54.119598 (+ 47us) server_negotiation.cc:407] Connection header received
0926 21:39:54.119664 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:54.119669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:54.119763 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:54.119862 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:54.121085 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:54.122233 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:54.123505 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:54.124111 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:54.124241 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:54.127005 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:54.127024 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:54.127044 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:54.127083 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:54.130544 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:54.131180 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:54.131190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:54.131193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:54.131356 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:54.131687 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:54.131692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:54.131695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:54.132131 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:54.132423 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:54.132633 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 21:39:54.132839 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:39:54.133944 13723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:54.225373 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 13597
I20260926 21:39:54.247524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 13219
I20260926 21:39:54.248906 13282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:54.363979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 13219
I20260926 21:39:54.393504 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 21:39:54.729043 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 21:39:54.729310 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 21:39:55.075133 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 21:39:55.075 UTC [12271] LOG: received smart shutdown request
2026-09-26 21:39:55.079 UTC [12271] LOG: background worker "logical replication launcher" (PID 12280) exited with exit code 1
2026-09-26 21:39:55.079 UTC [12275] LOG: shutting down
2026-09-26 21:39:55.243 UTC [12271] LOG: database system is shut down
2026-09-26T21:39:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54001 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:39:55 dist-test-slave-lhhq krb5kdc[13795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:39:55 dist-test-slave-lhhq krb5kdc[13795](info): set up 2 sockets
Sep 26 21:39:55 dist-test-slave-lhhq krb5kdc[13795](info): commencing operation
krb5kdc: starting...
W20260926 21:39:56.477551 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:39:56 dist-test-slave-lhhq krb5kdc[13795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:39:56.703085 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35105
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35105,127.0.97.61:35913,127.0.97.60:35967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:39:57.044006 13808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:57.044306 13808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:57.044466 13808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:57.051738 13808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:39:57.051851 13808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:57.051946 13808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:57.052047 13808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:39:57.052132 13808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:39:57.062075 13808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35105,127.0.97.61:35913,127.0.97.60:35967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:57.063457 13808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:57.064769 13808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:57.076984 13813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:57.076897 13814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:57.076866 13816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:57.077581 13808 server_base.cc:1034] running on GCE node
I20260926 21:39:57.078656 13808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:57.080168 13808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:57.081418 13808 hybrid_clock.cc:648] HybridClock initialized: now 1790458797081372 us; error 38 us; skew 500 ppm
I20260926 21:39:57.081784 13808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:57 dist-test-slave-lhhq krb5kdc[13795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:57.084945 13808 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:39:57.086503 13808 webserver.cc:460] Webserver started at http://127.0.97.62:39545/ using document root <none> and password file <none>
I20260926 21:39:57.087195 13808 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:57.087324 13808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:39:57.087638 13808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:39:57.090646 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "58e19899c10345fabf847e459e7d5989"
format_stamp: "Formatted at 2026-09-26 21:39:57 on dist-test-slave-lhhq"
server_key: "076afb1f7da3361d5398b086f12e953b"
server_key_iv: "f1ac2ddadbc22a5ad8de6cda7926e3ea"
server_key_version: "encryptionkey@0"
I20260926 21:39:57.091532 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "58e19899c10345fabf847e459e7d5989"
format_stamp: "Formatted at 2026-09-26 21:39:57 on dist-test-slave-lhhq"
server_key: "076afb1f7da3361d5398b086f12e953b"
server_key_iv: "f1ac2ddadbc22a5ad8de6cda7926e3ea"
server_key_version: "encryptionkey@0"
I20260926 21:39:57.096714 13808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 21:39:57.100337 13823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:57.101890 13808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:39:57.102092 13808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "58e19899c10345fabf847e459e7d5989"
format_stamp: "Formatted at 2026-09-26 21:39:57 on dist-test-slave-lhhq"
server_key: "076afb1f7da3361d5398b086f12e953b"
server_key_iv: "f1ac2ddadbc22a5ad8de6cda7926e3ea"
server_key_version: "encryptionkey@0"
I20260926 21:39:57.102311 13808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:57.125298 13808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:57.130019 13808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:57.130316 13808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:57.149047 13808 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35105
I20260926 21:39:57.149066 13874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35105 every 8 connection(s)
I20260926 21:39:57.150483 13808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:39:57.154181 13875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:39:57.154999 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 13808
I20260926 21:39:57.155239 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:39:57.156057 388 external_mini_cluster.cc:1428] Setting key 2d40d13557891c3779b29aacdb04bf11
I20260926 21:39:57.164726 13875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 26 21:39:57 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:39:57.189465 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.168116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35105 (local address 127.0.0.1:56990)
0926 21:39:57.169036 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.169081 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:57.171126 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:57.171981 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:57.171988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:57.172715 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:57.173060 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:57.173079 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.174757 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.174763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:57.175835 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:57.175846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.176759 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.176766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:57.176971 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:57.177681 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:57.177706 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:57.181868 (+ 4162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:57.185127 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:57.185144 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:57.185159 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:57.185786 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:57.186180 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:57.186187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:57.186194 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:57.186425 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:57.187198 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:57.187208 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:57.187481 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 21:39:57.187792 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 21:39:57.188304 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":1,"thread_start_us":411,"threads_started":1}
I20260926 21:39:57.189492 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.168116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.0.97.62:35105)
0926 21:39:57.169907 (+ 1791us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:57.169916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:57.169969 (+ 53us) server_negotiation.cc:407] Connection header received
0926 21:39:57.171517 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:57.171522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:57.171751 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:57.171969 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:57.173287 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.174604 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.175982 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.176535 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.176729 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:57.182110 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:57.182118 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:57.182131 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:57.182166 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:57.184977 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:57.185910 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:57.185916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:57.185919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:57.186066 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:57.186542 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:57.186545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:57.186547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:57.187049 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:57.187207 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:57.187967 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 21:39:57.188495 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":1575,"spinlock_wait_cycles":19200,"thread_start_us":1338,"threads_started":1}
I20260926 21:39:57.192536 13875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:39:57.193768 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.193392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37436)
0926 21:39:57.193550 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.193651 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 21:39:57.194049 13824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
I20260926 21:39:57.195590 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.195252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37440)
0926 21:39:57.195391 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.195475 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 21:39:57.195799 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35913: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 21:39:57.208925 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35913
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35105,127.0.97.61:35913,127.0.97.60:35967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:39:57.245157 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } attempt: 1
I20260926 21:39:57.246430 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.246005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37442)
0926 21:39:57.246198 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.246289 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 21:39:57.247424 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.247079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37458)
0926 21:39:57.247233 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.247299 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 21:39:57.247604 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35913: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
I20260926 21:39:57.315922 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } attempt: 2
I20260926 21:39:57.317088 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.316609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37470)
0926 21:39:57.316860 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.316951 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 21:39:57.318300 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.317762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37472)
0926 21:39:57.318000 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.318106 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 21:39:57.318522 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35913: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
I20260926 21:39:57.409798 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } attempt: 3
I20260926 21:39:57.411036 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.410518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37476)
0926 21:39:57.410749 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.410863 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 21:39:57.412170 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.411749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37484)
0926 21:39:57.411913 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.411999 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 21:39:57.412431 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35913: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
W20260926 21:39:57.540184 13882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:57.540488 13882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:57.540679 13882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:57.548161 13882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:39:57.548293 13882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:57.548375 13882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:57.548461 13882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:39:57.548519 13882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:39:57.555707 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } attempt: 4
I20260926 21:39:57.557052 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.556408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37496)
0926 21:39:57.556757 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.556871 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260926 21:39:57.558282 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.557808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37508)
0926 21:39:57.557993 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.558102 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 21:39:57.558491 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35913: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35913: connect: Connection refused (error 111)
I20260926 21:39:57.559490 13882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35105,127.0.97.61:35913,127.0.97.60:35967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:57.560894 13882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:57.562336 13882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:57.574739 13887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:57.574846 13890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:57.574774 13888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:57.575464 13882 server_base.cc:1034] running on GCE node
I20260926 21:39:57.576447 13882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:57.578001 13882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:57.579231 13882 hybrid_clock.cc:648] HybridClock initialized: now 1790458797579176 us; error 51 us; skew 500 ppm
I20260926 21:39:57.579593 13882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:57 dist-test-slave-lhhq krb5kdc[13795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:57.582751 13882 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 21:39:57.584219 13882 webserver.cc:460] Webserver started at http://127.0.97.61:46249/ using document root <none> and password file <none>
I20260926 21:39:57.584939 13882 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:57.585083 13882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:39:57.585464 13882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:39:57.588515 13882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/data/instance:
uuid: "5cc05c37211246479dde699c8081b2cf"
format_stamp: "Formatted at 2026-09-26 21:39:57 on dist-test-slave-lhhq"
server_key: "f3873cc5598ac511e64bb5dda267fd87"
server_key_iv: "24b7bf3f5433afd09fc364100439568b"
server_key_version: "encryptionkey@0"
I20260926 21:39:57.589378 13882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/wal/instance:
uuid: "5cc05c37211246479dde699c8081b2cf"
format_stamp: "Formatted at 2026-09-26 21:39:57 on dist-test-slave-lhhq"
server_key: "f3873cc5598ac511e64bb5dda267fd87"
server_key_iv: "24b7bf3f5433afd09fc364100439568b"
server_key_version: "encryptionkey@0"
I20260926 21:39:57.594893 13882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:39:57.598609 13897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:57.600018 13882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:39:57.600200 13882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/wal
uuid: "5cc05c37211246479dde699c8081b2cf"
format_stamp: "Formatted at 2026-09-26 21:39:57 on dist-test-slave-lhhq"
server_key: "f3873cc5598ac511e64bb5dda267fd87"
server_key_iv: "24b7bf3f5433afd09fc364100439568b"
server_key_version: "encryptionkey@0"
I20260926 21:39:57.600410 13882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:57.618347 13882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:57.622732 13882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:57.623091 13882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:57.642392 13882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35913
I20260926 21:39:57.642417 13948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35913 every 8 connection(s)
I20260926 21:39:57.643841 13882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:39:57.647495 13949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:39:57.652496 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 13882
I20260926 21:39:57.652699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-1/wal/instance
I20260926 21:39:57.653234 388 external_mini_cluster.cc:1428] Setting key d9ad16ef73a0ef3bcc619ff7884dd7ad
I20260926 21:39:57.654546 13949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:39:57 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260926 21:39:57.678670 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.658012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.0.97.62:35105)
0926 21:39:57.658266 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:57.658275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:57.659382 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 21:39:57.661231 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:57.661237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:57.661330 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:57.661463 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:57.663439 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.664122 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.666647 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.667103 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.667201 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:57.671600 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:57.671609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:57.671617 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:57.671672 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:57.675450 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:57.676283 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:57.676288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:57.676292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:57.676452 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:57.676913 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:57.676917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:57.676919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:57.677258 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:57.677417 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:57.678239 (+ 822us) server_negotiation.cc:299] Negotiation successful
0926 21:39:57.678434 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 21:39:57.680078 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.658394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35105 (local address 127.0.0.1:56998)
0926 21:39:57.659195 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.659249 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:57.660935 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:57.661698 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:57.661711 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:57.662480 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:57.663257 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:57.663283 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.664299 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.664307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:57.666486 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:57.666509 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.667348 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.667358 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:57.667690 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:57.668456 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:57.668491 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:57.671263 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:57.675628 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:57.675636 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:57.675651 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:57.676147 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:57.676590 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:57.676596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:57.676599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:57.676784 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:57.677401 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:57.677408 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:57.677863 (+ 455us) client_negotiation.cc:769] Sending connection context
0926 21:39:57.678111 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 21:39:57.678617 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":345,"threads_started":1}
I20260926 21:39:57.681921 13949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:39:57 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:39:57.697641 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.682716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37512)
0926 21:39:57.682927 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.682944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:57.683031 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:57.684195 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:57.684200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:57.684675 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:57.684852 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:57.684861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.686354 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.686362 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:57.687252 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:57.687263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.688104 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.688110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:57.688346 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:57.688997 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:57.689016 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:57.691344 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:57.694792 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:57.694799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:57.694803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:57.695196 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:57.695613 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:57.695619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:57.695623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:57.695779 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:57.696443 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:57.696449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:57.696605 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 21:39:57.696769 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 21:39:57.697063 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 21:39:57.697687 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.683031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.0.97.61:35913)
0926 21:39:57.683753 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:57.683759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:57.683788 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:39:57.683833 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:57.683839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:57.684022 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:57.684226 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:57.685031 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.686189 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.687396 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.687950 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.688097 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:57.691569 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:57.691580 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:57.691593 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:57.691622 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:57.694620 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:57.695335 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:57.695345 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:57.695349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:57.695477 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:57.695900 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:57.695908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:57.695916 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:57.696295 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:57.696527 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:57.696908 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 21:39:57.697208 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":540,"thread_start_us":404,"threads_started":1}
I20260926 21:39:57.700187 13949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:39:57.701387 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.700850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43450)
0926 21:39:57.701022 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.701233 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 21:39:57.701761 13898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
I20260926 21:39:57.703349 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.703009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43462)
0926 21:39:57.703170 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.703248 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 21:39:57.703562 13949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35967: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 21:39:57.708441 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35967
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35105,127.0.97.61:35913,127.0.97.60:35967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:39:57.752907 13949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } attempt: 1
I20260926 21:39:57.754163 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.753693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43474)
0926 21:39:57.753901 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.754012 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 21:39:57.755213 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.754866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43488)
0926 21:39:57.755033 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.755103 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 21:39:57.755478 13949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35967: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
I20260926 21:39:57.823763 13949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } attempt: 2
I20260926 21:39:57.824977 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.824493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43494)
0926 21:39:57.824696 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.824804 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 21:39:57.826057 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.825708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43498)
0926 21:39:57.825870 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.825939 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 21:39:57.826323 13949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35967: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
I20260926 21:39:57.857795 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } attempt: 5
Sep 26 21:39:57 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:39:57.872339 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.858637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37514)
0926 21:39:57.858855 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.858905 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:57.859054 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:57.859442 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:57.859451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:57.859927 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:57.860124 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:57.860134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.861287 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.861300 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:57.862183 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:57.862196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.862983 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.862990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:57.863210 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:57.863956 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:57.863975 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:57.866316 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:57.869632 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:57.869639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:57.869644 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:57.870047 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:57.870462 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:57.870468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:57.870472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:57.870629 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:57.871234 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:57.871240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:57.871411 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 21:39:57.871571 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 21:39:57.871907 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 21:39:57.872334 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.858841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.0.97.61:35913)
0926 21:39:57.859068 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:57.859075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:57.859100 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:39:57.859184 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:57.859191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:57.859272 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:57.859394 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:57.860298 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.861077 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.862342 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:57.862819 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:57.862979 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:57.866539 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:57.866545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:57.866548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:57.866592 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:57.869479 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:57.870202 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:57.870208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:57.870210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:57.870340 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:57.870759 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:57.870767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:57.870771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:57.871081 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:57.871257 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:57.871694 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 21:39:57.871919 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:39:57.873663 13875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:39:57.875036 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.874434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43514)
0926 21:39:57.874657 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.874777 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 21:39:57.876221 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.875743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43526)
0926 21:39:57.875925 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.876052 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 21:39:57.876456 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35967: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
I20260926 21:39:57.917572 13949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } attempt: 3
I20260926 21:39:57.918769 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.918304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43542)
0926 21:39:57.918519 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.918609 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 21:39:57.919930 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.919509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43554)
0926 21:39:57.919730 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.919807 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260926 21:39:57.920138 13949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35967: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
I20260926 21:39:57.927723 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } attempt: 1
I20260926 21:39:57.928970 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.928423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43568)
0926 21:39:57.928640 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.928757 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 21:39:57.929991 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.929644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43578)
0926 21:39:57.929787 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.929866 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 21:39:57.930212 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35967: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
I20260926 21:39:57.998452 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } attempt: 2
I20260926 21:39:57.999758 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:57.999152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43582)
0926 21:39:57.999402 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:57.999542 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 21:39:58.000818 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.000429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43586)
0926 21:39:58.000596 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.000689 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 21:39:58.001075 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35967: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
W20260926 21:39:58.050800 13956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:39:58.051092 13956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:39:58.051183 13956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:39:58.058388 13956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:39:58.058504 13956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:39:58.058552 13956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:39:58.058610 13956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:39:58.058655 13956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:39:58.063354 13949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } attempt: 4
I20260926 21:39:58.064615 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.064093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43590)
0926 21:39:58.064343 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.064445 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 21:39:58.065728 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.065376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43602)
0926 21:39:58.065527 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.065606 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:39:58.066012 13949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35967: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
I20260926 21:39:58.068898 13956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35105,127.0.97.61:35913,127.0.97.60:35967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:39:58.070312 13956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:39:58.071687 13956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:39:58.084211 13964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:58.084317 13962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:39:58.084765 13961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:39:58.085000 13956 server_base.cc:1034] running on GCE node
I20260926 21:39:58.085924 13956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:39:58.087417 13956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:39:58.088651 13956 hybrid_clock.cc:648] HybridClock initialized: now 1790458798088603 us; error 71 us; skew 500 ppm
I20260926 21:39:58.089020 13956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:39:58 dist-test-slave-lhhq krb5kdc[13795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:39:58.092227 13956 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 21:39:58.093770 13956 webserver.cc:460] Webserver started at http://127.0.97.60:36905/ using document root <none> and password file <none>
I20260926 21:39:58.094496 13956 fs_manager.cc:359] Metadata directory not provided
I20260926 21:39:58.094645 13956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:39:58.094957 13956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:39:58.098109 13956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/data/instance:
uuid: "fbe939ee80b7449689ff7942689db167"
format_stamp: "Formatted at 2026-09-26 21:39:58 on dist-test-slave-lhhq"
server_key: "6113782f7414d34526aa67ddeddd5b1a"
server_key_iv: "a8932eae91c5da42c3ef08f09bfb98c5"
server_key_version: "encryptionkey@0"
I20260926 21:39:58.098907 13956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/wal/instance:
uuid: "fbe939ee80b7449689ff7942689db167"
format_stamp: "Formatted at 2026-09-26 21:39:58 on dist-test-slave-lhhq"
server_key: "6113782f7414d34526aa67ddeddd5b1a"
server_key_iv: "a8932eae91c5da42c3ef08f09bfb98c5"
server_key_version: "encryptionkey@0"
I20260926 21:39:58.104269 13956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:39:58.107300 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } attempt: 3
I20260926 21:39:58.108479 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.107887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43604)
0926 21:39:58.108189 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.108310 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 21:39:58.108500 13971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:39:58.109836 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.109357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43606)
0926 21:39:58.109627 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.109713 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260926 21:39:58.110071 13875 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35967: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35967: connect: Connection refused (error 111)
I20260926 21:39:58.110402 13956 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 21:39:58.110570 13956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/wal
uuid: "fbe939ee80b7449689ff7942689db167"
format_stamp: "Formatted at 2026-09-26 21:39:58 on dist-test-slave-lhhq"
server_key: "6113782f7414d34526aa67ddeddd5b1a"
server_key_iv: "a8932eae91c5da42c3ef08f09bfb98c5"
server_key_version: "encryptionkey@0"
I20260926 21:39:58.110744 13956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:39:58.123888 13956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:39:58.128016 13956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:39:58.128336 13956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:39:58.146994 13956 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35967
I20260926 21:39:58.147018 14022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35967 every 8 connection(s)
I20260926 21:39:58.148402 13956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:39:58.152302 14023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:39:58.154011 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 13956
I20260926 21:39:58.154213 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790458593267894-388-0/minicluster-data/master-2/wal/instance
I20260926 21:39:58.154750 388 external_mini_cluster.cc:1428] Setting key 4b3952055e3ef96f0c804df7c7f77130
I20260926 21:39:58.160522 14023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:39:58 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:39:58 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:39:58.177824 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.156635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.0.97.60:35967)
0926 21:39:58.157680 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.157697 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.158067 (+ 370us) server_negotiation.cc:407] Connection header received
0926 21:39:58.159880 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.159901 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.160269 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.160643 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.162229 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.163728 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.165390 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.166103 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.166301 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.170388 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.170401 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.170417 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.170469 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.173743 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.174447 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.174454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.174461 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.174597 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.174993 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.175002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.175004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.175604 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.175898 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.176311 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.176721 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":538,"threads_started":1}
I20260926 21:39:58.178583 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.163008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.0.97.62:35105)
0926 21:39:58.163226 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.163231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.164203 (+ 972us) server_negotiation.cc:407] Connection header received
0926 21:39:58.164407 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.164412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.164492 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.164601 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.165847 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.166604 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.168017 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.168491 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.168631 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.172559 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.172564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.172568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.172604 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.175430 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.176302 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.176309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.176312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.176483 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.176922 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.176929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.176932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.177269 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.177434 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.177943 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.178169 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:39:58.178650 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.163115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35105 (local address 127.0.0.1:57012)
0926 21:39:58.164054 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.164097 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:58.164267 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:58.164662 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:58.164667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:58.165362 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:58.165672 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.165682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.166772 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.166780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.167854 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.167866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.168746 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.168755 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.168973 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.169735 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:58.169760 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:58.172371 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:58.175556 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.175568 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.175574 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.176161 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.176635 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.176641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.176645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.176798 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.177414 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:58.177424 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:58.177624 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 21:39:58.177843 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 21:39:58.178154 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":482,"threads_started":1}
I20260926 21:39:58.180195 14023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:39:58 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 21:39:58 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:39:58.194306 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.181000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35913 (local address 127.0.0.1:37528)
0926 21:39:58.181211 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.181231 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:58.181354 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:58.181789 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:58.181794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:58.182289 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:58.182503 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.182516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.183486 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.183496 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.184651 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.184668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.185407 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.185414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.185639 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.186217 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:58.186244 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:58.188490 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:58.192047 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.192050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.192053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.192413 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.192815 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.192823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.192826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.192971 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.193579 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:58.193582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:58.193733 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 21:39:58.193848 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 21:39:58.194045 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 21:39:58.194589 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.181016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.0.97.61:35913)
0926 21:39:58.181226 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.181234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.181278 (+ 44us) server_negotiation.cc:407] Connection header received
0926 21:39:58.181551 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.181557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.181638 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.181736 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.182664 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.183335 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.184800 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.185245 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.185350 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.188684 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.188691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.188695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.188735 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.191917 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.192513 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.192518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.192521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.192672 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.193092 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.193099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.193105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.193456 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.193617 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.193982 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.194205 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:39:58.195642 14023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:39:58.198041 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.181537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.0.97.62:35105)
0926 21:39:58.181689 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.181696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.181887 (+ 191us) server_negotiation.cc:407] Connection header received
0926 21:39:58.182053 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.182057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.182107 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.182178 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.183569 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.184272 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.186236 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.186617 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.186705 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.191565 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.191572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.191577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.191626 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.195002 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.195903 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.195909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.195912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.196043 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.196578 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.196583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.196586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.196881 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.197216 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.197499 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.197720 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 21:39:58 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:39:58 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:39:58.209758 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.196578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.0.97.60:35967)
0926 21:39:58.196749 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.196754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.196768 (+ 14us) server_negotiation.cc:407] Connection header received
0926 21:39:58.196810 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.196813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.196861 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.196950 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.197662 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.198510 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.199892 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.200442 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.200572 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.204370 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.204377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.204381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.204420 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.207238 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.207873 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.207881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.207884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.207992 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.208394 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.208398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.208400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.208730 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.208888 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.209305 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.209483 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:39:58.210130 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.196358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43628)
0926 21:39:58.196521 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.196540 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:58.196660 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:58.196964 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:58.196968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:58.197417 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:58.197537 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.197545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.198622 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.198630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.199706 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.199727 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.200592 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.200597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.200788 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.201484 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:58.201502 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:58.204154 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:58.207390 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.207395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.207399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.207742 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.208092 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.208098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.208102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.208270 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.208856 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:58.208861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:58.209024 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 21:39:58.209209 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 21:39:58.209697 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":100}
I20260926 21:39:58.215314 14023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167: Bootstrap starting.
I20260926 21:39:58.215704 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.199922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.97.61:35913)
0926 21:39:58.200086 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.200091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.200108 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:39:58.200155 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.200158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.200232 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.200320 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.201751 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.202913 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.204492 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.205013 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.205162 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.210021 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.210027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.210031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.210073 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.212865 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.213653 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.213659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.213663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.213830 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.214226 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.214231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.214234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.214574 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.214868 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.215160 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.215380 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 21:39:58.219368 14023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167: Neither blocks nor log segments found. Creating new log.
I20260926 21:39:58.220553 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:39:58.220760 14023 log.cc:824] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167: Log is configured to *not* fsync() on all Append() calls
I20260926 21:39:58.224071 14023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167: No bootstrap required, opened a new log
I20260926 21:39:58.236667 14023 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } }
I20260926 21:39:58.237407 14023 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:39:58.237607 14023 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe939ee80b7449689ff7942689db167, State: Initialized, Role: FOLLOWER
I20260926 21:39:58.238560 14023 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } }
I20260926 21:39:58.243903 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.244397 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:39:58.246644 14023 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:39:58.249008 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.226080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.0.97.62:35105)
0926 21:39:58.226254 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.226260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.230009 (+ 3749us) server_negotiation.cc:407] Connection header received
0926 21:39:58.230219 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.230224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.230318 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.230426 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.232778 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.233949 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.236157 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.236803 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.236918 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.241234 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.241241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.241248 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.241300 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.245298 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.246424 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.246430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.246434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.246610 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.247107 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.247112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.247115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.247475 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.247701 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.248342 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.248594 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:39:58.251845 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.225667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.0.97.61:35913)
0926 21:39:58.225883 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.225892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.230756 (+ 4864us) server_negotiation.cc:407] Connection header received
0926 21:39:58.231030 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.231035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.231112 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.231183 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.234266 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.235740 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.238562 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.239465 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.239561 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.243893 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.243901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.243904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.243949 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.247494 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.248636 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.248645 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.248651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.248873 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.249517 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.249525 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.249531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.250026 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.250331 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.251018 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.251331 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:39:58.257273 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.230928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.0.97.60:35967)
0926 21:39:58.231113 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.231118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.239857 (+ 8739us) server_negotiation.cc:407] Connection header received
0926 21:39:58.240173 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.240179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.240266 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.240352 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.242595 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.243803 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.246217 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.247018 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.247123 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.250522 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.250530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.250533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.250577 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.254083 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.254898 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.254901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.254904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.255035 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.255480 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.255491 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.255496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.256006 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.256324 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.256639 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.256875 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 21:39:58.262521 14047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:39:58.262732 14047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:39:58.259685 13956 master_runner.cc:186] Error getting master registration for 127.0.97.62:35105: OK, instance_id { permanent_uuid: "58e19899c10345fabf847e459e7d5989" instance_seqno: 1790458797143617 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:39:58.268056 13956 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:39:58.287300 13875 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } attempt: 4
Sep 26 21:39:58 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:39:58.302387 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.288002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.0.97.60:35967)
0926 21:39:58.288156 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.288160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.288177 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:39:58.288369 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.288372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.288429 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.288506 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.289477 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.290255 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.292035 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.292475 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.292567 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.296764 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.296770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.296776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.296834 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.299656 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.300424 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.300428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.300431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.300574 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.301005 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.301008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.301010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.301369 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.301498 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.301949 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.302113 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 21:39:58.302588 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.287901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:43650)
0926 21:39:58.288095 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.288118 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:58.288233 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:58.288567 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:58.288574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:58.289100 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:58.289338 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.289351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.290408 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.290414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.291890 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.291907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.292617 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.292624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.292818 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.293500 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:58.293525 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:58.296506 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:58.299790 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.299796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.299802 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.300311 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.300688 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.300693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.300697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.300894 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.301504 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:58.301510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:58.301691 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 21:39:58.301884 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 21:39:58.302205 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 21:39:58.307930 13875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989: Bootstrap starting.
I20260926 21:39:58.310995 13875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989: Neither blocks nor log segments found. Creating new log.
I20260926 21:39:58.312237 13875 log.cc:824] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989: Log is configured to *not* fsync() on all Append() calls
I20260926 21:39:58.315307 13875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989: No bootstrap required, opened a new log
I20260926 21:39:58.323335 13875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } }
I20260926 21:39:58.323724 13875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:39:58.323848 13875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e19899c10345fabf847e459e7d5989, State: Initialized, Role: FOLLOWER
I20260926 21:39:58.324424 13875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } }
I20260926 21:39:58.325390 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.325759 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:39:58.326550 13875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:39:58.338230 14061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:39:58.338372 14061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:39:58.335608 13808 master_runner.cc:186] Error getting master registration for 127.0.97.61:35913: OK, instance_id { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" instance_seqno: 1790458797636620 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:39:58.348470 13808 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:39:58.365243 13949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } attempt: 5
Sep 26 21:39:58 dist-test-slave-lhhq krb5kdc[13795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260926 21:39:58.380779 13977 tablet.cc:2369] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:39:58.383180 13829 tablet.cc:2369] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:39:58.386575 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.365916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35967 (local address 127.0.0.1:50982)
0926 21:39:58.366286 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 21:39:58.366306 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:39:58.366413 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:39:58.367778 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:39:58.367783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:39:58.368334 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:39:58.368553 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.368566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.370073 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.370079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.371551 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:39:58.371568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.372606 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.372612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:39:58.372818 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.373627 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:39:58.373653 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:39:58.376534 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:39:58.383616 (+ 7082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.383622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.383626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.384190 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.384615 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:39:58.384621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:39:58.384624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:39:58.384799 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:39:58.385509 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:39:58.385514 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:39:58.385689 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 21:39:58.385867 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 21:39:58.386168 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260926 21:39:58.386651 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:39:58.367205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.0.97.60:35967)
0926 21:39:58.367459 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 21:39:58.367467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:39:58.367486 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:39:58.367537 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:39:58.367542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:39:58.367612 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:39:58.367705 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:39:58.368726 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.369911 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.371713 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:39:58.372436 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:39:58.372578 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:39:58.376805 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:39:58.376815 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:39:58.376821 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:39:58.376870 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:39:58.383451 (+ 6581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.384329 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.384336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.384341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.384489 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:39:58.384932 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:39:58.384938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:39:58.384942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:39:58.385381 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:39:58.385565 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 21:39:58.385975 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 21:39:58.386234 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260926 21:39:58.392992 13949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf: Bootstrap starting.
I20260926 21:39:58.397174 13949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf: Neither blocks nor log segments found. Creating new log.
I20260926 21:39:58.398694 13949 log.cc:824] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf: Log is configured to *not* fsync() on all Append() calls
I20260926 21:39:58.402938 13949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf: No bootstrap required, opened a new log
I20260926 21:39:58.414206 13949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } }
I20260926 21:39:58.414649 13949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:39:58.414791 13949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc05c37211246479dde699c8081b2cf, State: Initialized, Role: FOLLOWER
I20260926 21:39:58.415486 13949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } }
I20260926 21:39:58.419691 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.420183 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:39:58.421043 13949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:39:58.441817 13882 master_runner.cc:263] No leader master found from master 5cc05c37211246479dde699c8081b2cf
I20260926 21:39:58.441991 13882 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 21:39:58.445056 14075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:39:58.445338 14075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:39:58.597635 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:39:58.597965 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } }
I20260926 21:39:58.599637 14064 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58e19899c10345fabf847e459e7d5989 (127.0.97.62:35105), fbe939ee80b7449689ff7942689db167 (127.0.97.60:35967)
I20260926 21:39:58.600888 13998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cc05c37211246479dde699c8081b2cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe939ee80b7449689ff7942689db167" is_pre_election: true
I20260926 21:39:58.601636 13998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cc05c37211246479dde699c8081b2cf in term 0.
I20260926 21:39:58.603058 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cc05c37211246479dde699c8081b2cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e19899c10345fabf847e459e7d5989" is_pre_election: true
I20260926 21:39:58.603700 13850 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cc05c37211246479dde699c8081b2cf in term 0.
I20260926 21:39:58.604661 13898 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc05c37211246479dde699c8081b2cf, fbe939ee80b7449689ff7942689db167; no voters:
I20260926 21:39:58.605299 14064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:39:58.605502 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:39:58.605623 14064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 0 FOLLOWER]: Advancing to term 1
W20260926 21:39:58.625797 13903 tablet.cc:2369] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:39:58.612610 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } }
I20260926 21:39:58.628089 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cc05c37211246479dde699c8081b2cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e19899c10345fabf847e459e7d5989"
I20260926 21:39:58.628394 13850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:39:58.632112 13850 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cc05c37211246479dde699c8081b2cf in term 1.
I20260926 21:39:58.632882 13900 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e19899c10345fabf847e459e7d5989, 5cc05c37211246479dde699c8081b2cf; no voters:
I20260926 21:39:58.634966 13998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cc05c37211246479dde699c8081b2cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe939ee80b7449689ff7942689db167"
I20260926 21:39:58.635231 13998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:39:58.638823 13998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cc05c37211246479dde699c8081b2cf in term 1.
I20260926 21:39:58.639767 14064 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [CANDIDATE]: Term 1 election: Requested vote from peers 58e19899c10345fabf847e459e7d5989 (127.0.97.62:35105), fbe939ee80b7449689ff7942689db167 (127.0.97.60:35967)
I20260926 21:39:58.640278 14064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:39:58.640770 14064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [term 1 LEADER]: Becoming Leader. State: Replica: 5cc05c37211246479dde699c8081b2cf, State: Running, Role: LEADER
I20260926 21:39:58.642851 14064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } }
I20260926 21:39:58.683216 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc05c37211246479dde699c8081b2cf. Latest consensus state: current_term: 1 leader_uuid: "5cc05c37211246479dde699c8081b2cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.683809 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [sys.catalog]: This master's current role is: LEADER
I20260926 21:39:58.689805 14079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:39:58.692042 14079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:39:58.727790 13850 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [term 1 FOLLOWER]: Refusing update from remote peer 5cc05c37211246479dde699c8081b2cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:39:58.729172 13998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [term 1 FOLLOWER]: Refusing update from remote peer 5cc05c37211246479dde699c8081b2cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:39:58.730491 14077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:39:58.732095 14077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:39:58.765853 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc05c37211246479dde699c8081b2cf. Latest consensus state: current_term: 1 leader_uuid: "5cc05c37211246479dde699c8081b2cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.766152 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:39:58.770992 14080 mvcc.cc:205] Tried to move back new op lower bound from 7333719239576174592 to 7333719239334105088. Current Snapshot: MvccSnapshot[applied={T|T < 7333719239576174592}]
I20260926 21:39:58.771481 14081 mvcc.cc:205] Tried to move back new op lower bound from 7333719239576174592 to 7333719239334105088. Current Snapshot: MvccSnapshot[applied={T|T < 7333719239576174592}]
I20260926 21:39:58.785455 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cc05c37211246479dde699c8081b2cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.785893 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [sys.catalog]: This master's current role is: LEADER
I20260926 21:39:58.791535 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cc05c37211246479dde699c8081b2cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.791508 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc05c37211246479dde699c8081b2cf. Latest consensus state: current_term: 1 leader_uuid: "5cc05c37211246479dde699c8081b2cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.791793 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:39:58.791846 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:39:58.799037 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cc05c37211246479dde699c8081b2cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.799929 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf [sys.catalog]: This master's current role is: LEADER
I20260926 21:39:58.806674 14079 catalog_manager.cc:1347] Generated new cluster ID: f4451b0990464a9c9b4f6268a10d75a2
I20260926 21:39:58.806815 14079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:39:58.806957 14082 mvcc.cc:205] Tried to move back new op lower bound from 7333719239576174592 to 7333719239334105088. Current Snapshot: MvccSnapshot[applied={T|T < 7333719239576174592}]
I20260926 21:39:58.811058 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cc05c37211246479dde699c8081b2cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e19899c10345fabf847e459e7d5989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "5cc05c37211246479dde699c8081b2cf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35913 } } peers { permanent_uuid: "fbe939ee80b7449689ff7942689db167" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35967 } } }
I20260926 21:39:58.811326 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:39:58.834198 14079 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:39:58.836249 14079 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:39:58.857481 14079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cc05c37211246479dde699c8081b2cf: Generated new TSK 0
I20260926 21:39:58.858709 14079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:39:59.264746 14047 catalog_manager.cc:1259] Loaded cluster ID: f4451b0990464a9c9b4f6268a10d75a2
I20260926 21:39:59.264864 14047 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167: loading cluster ID for follower catalog manager: success
I20260926 21:39:59.267314 14047 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167: acquiring CA information for follower catalog manager: success
I20260926 21:39:59.268989 14047 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbe939ee80b7449689ff7942689db167: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:39:59.340322 14061 catalog_manager.cc:1259] Loaded cluster ID: f4451b0990464a9c9b4f6268a10d75a2
I20260926 21:39:59.340444 14061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989: loading cluster ID for follower catalog manager: success
I20260926 21:39:59.343022 14061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989: acquiring CA information for follower catalog manager: success
I20260926 21:39:59.344765 14061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 58e19899c10345fabf847e459e7d5989: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:39:59.440208 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 13808
I20260926 21:39:59.442085 13870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:59.552798 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 13808
I20260926 21:39:59.573814 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 13882
I20260926 21:39:59.574911 13944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:59.685577 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 13882
I20260926 21:39:59.706429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 13956
I20260926 21:39:59.707522 14018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:39:59.817378 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 13956
2026-09-26T21:39:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4448 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:39:59 dist-test-slave-lhhq krb5kdc[14098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:39:59 dist-test-slave-lhhq krb5kdc[14098](info): set up 2 sockets
Sep 26 21:39:59 dist-test-slave-lhhq krb5kdc[14098](info): commencing operation
krb5kdc: starting...
W20260926 21:40:00.925719 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:40:01 dist-test-slave-lhhq krb5kdc[14098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:40:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:40:01.161788 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32821
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:01.491210 14111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:01.491581 14111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:01.491742 14111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:01.498965 14111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:40:01.499117 14111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:01.499243 14111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:01.499353 14111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:40:01.499423 14111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:40:01.509436 14111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:01.510783 14111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:01.512130 14111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:01.524279 14116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:01.524240 14117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:01.524535 14119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:01.525178 14111 server_base.cc:1034] running on GCE node
I20260926 21:40:01.526017 14111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:01.527341 14111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:01.528549 14111 hybrid_clock.cc:648] HybridClock initialized: now 1790458801528507 us; error 33 us; skew 500 ppm
I20260926 21:40:01.528847 14111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:01 dist-test-slave-lhhq krb5kdc[14098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:01.531963 14111 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:40:01.534096 14111 webserver.cc:460] Webserver started at http://127.0.97.62:33133/ using document root <none> and password file <none>
I20260926 21:40:01.534816 14111 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:01.534960 14111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:01.535298 14111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:40:01.538305 14111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "021741cfb0604d028d61a2e1eeac087f"
format_stamp: "Formatted at 2026-09-26 21:40:01 on dist-test-slave-lhhq"
server_key: "edd6ba6b9cda98461f98c300d3061f64"
server_key_iv: "4e356e1df3d96b9af94345cf201bfcbf"
server_key_version: "encryptionkey@0"
I20260926 21:40:01.539095 14111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "021741cfb0604d028d61a2e1eeac087f"
format_stamp: "Formatted at 2026-09-26 21:40:01 on dist-test-slave-lhhq"
server_key: "edd6ba6b9cda98461f98c300d3061f64"
server_key_iv: "4e356e1df3d96b9af94345cf201bfcbf"
server_key_version: "encryptionkey@0"
I20260926 21:40:01.544399 14111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:40:01.547824 14126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:01.549077 14111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:40:01.549311 14111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "021741cfb0604d028d61a2e1eeac087f"
format_stamp: "Formatted at 2026-09-26 21:40:01 on dist-test-slave-lhhq"
server_key: "edd6ba6b9cda98461f98c300d3061f64"
server_key_iv: "4e356e1df3d96b9af94345cf201bfcbf"
server_key_version: "encryptionkey@0"
I20260926 21:40:01.549515 14111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:01.571695 14111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:01.576371 14111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:01.576728 14111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:01.596999 14111 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32821
I20260926 21:40:01.597040 14177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32821 every 8 connection(s)
I20260926 21:40:01.598553 14111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:40:01.602488 14178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:01.604416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 14111
I20260926 21:40:01.604655 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:40:01.605562 388 external_mini_cluster.cc:1428] Setting key c7fc9041b6f0b26c35b2e92af92c354e
I20260926 21:40:01.617607 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f: Bootstrap starting.
Sep 26 21:40:01 dist-test-slave-lhhq krb5kdc[14098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:01.621645 14178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:01.622711 14178 log.cc:824] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:01.626648 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f: No bootstrap required, opened a new log
I20260926 21:40:01.631392 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:01.607493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.0.97.62:32821)
0926 21:40:01.608356 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:01.608377 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:01.608417 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:40:01.610079 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:01.610106 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:01.610520 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:01.610869 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:01.612774 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:01.614240 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:01.616152 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:01.616813 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:01.616953 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:01.622325 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:01.622334 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:01.622349 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:01.622393 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:01.626369 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:01.627206 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:01.627216 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:01.627225 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:01.627455 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:01.627880 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:01.627885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:01.627889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:01.628626 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:01.628992 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:01.629401 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 21:40:01.629884 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":418,"threads_started":1}
I20260926 21:40:01.635733 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:40:01.637357 14178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021741cfb0604d028d61a2e1eeac087f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } }
I20260926 21:40:01.637784 14178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:01.637923 14178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021741cfb0604d028d61a2e1eeac087f, State: Initialized, Role: FOLLOWER
I20260926 21:40:01.638530 14178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021741cfb0604d028d61a2e1eeac087f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } }
I20260926 21:40:01.638825 14178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:40:01.639007 14178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:40:01.639202 14178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:01.641810 14178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021741cfb0604d028d61a2e1eeac087f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } }
I20260926 21:40:01.642378 14178 leader_election.cc:302] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 021741cfb0604d028d61a2e1eeac087f; no voters:
I20260926 21:40:01.643064 14178 leader_election.cc:288] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:40:01.643327 14183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:40:01.644071 14183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [term 1 LEADER]: Becoming Leader. State: Replica: 021741cfb0604d028d61a2e1eeac087f, State: Running, Role: LEADER
I20260926 21:40:01.644651 14183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021741cfb0604d028d61a2e1eeac087f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } }
I20260926 21:40:01.645510 14178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:40:01.648185 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "021741cfb0604d028d61a2e1eeac087f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021741cfb0604d028d61a2e1eeac087f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } } }
I20260926 21:40:01.648644 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:01.650341 14192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:40:01.650733 14185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 021741cfb0604d028d61a2e1eeac087f. Latest consensus state: current_term: 1 leader_uuid: "021741cfb0604d028d61a2e1eeac087f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021741cfb0604d028d61a2e1eeac087f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } } }
I20260926 21:40:01.651122 14185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:01.657270 14192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:40:01.666262 14192 catalog_manager.cc:1347] Generated new cluster ID: a7f720e3b7e444a49e80d17936ad8cdf
I20260926 21:40:01.666386 14192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:40:01.710213 14192 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:40:01.711627 14192 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:40:01.719209 14192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 021741cfb0604d028d61a2e1eeac087f: Generated new TSK 0
I20260926 21:40:01.720221 14192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:40:01.748698 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 14111
I20260926 21:40:01.750068 14173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:01.848353 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 14111
2026-09-26T21:40:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2027 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:40:01 dist-test-slave-lhhq krb5kdc[14206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:40:01 dist-test-slave-lhhq krb5kdc[14206](info): set up 2 sockets
Sep 26 21:40:01 dist-test-slave-lhhq krb5kdc[14206](info): commencing operation
krb5kdc: starting...
W20260926 21:40:02.953894 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:40:03 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:40:03.195451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39563
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:03.529546 14219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:03.529858 14219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:03.530066 14219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:03.537496 14219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:40:03.537613 14219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:03.537729 14219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:03.537834 14219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:40:03.537918 14219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:40:03.548291 14219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:03.549790 14219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:03.551142 14219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:03.562925 14224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:03.563407 14225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:03.563443 14227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:03.565243 14219 server_base.cc:1034] running on GCE node
I20260926 21:40:03.566042 14219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:03.567536 14219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:03.568769 14219 hybrid_clock.cc:648] HybridClock initialized: now 1790458803568731 us; error 52 us; skew 500 ppm
I20260926 21:40:03.569120 14219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:03 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:03.572223 14219 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:40:03.573788 14219 webserver.cc:460] Webserver started at http://127.0.97.62:37651/ using document root <none> and password file <none>
I20260926 21:40:03.574484 14219 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:03.574623 14219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:03.574925 14219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:40:03.577909 14219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f"
format_stamp: "Formatted at 2026-09-26 21:40:03 on dist-test-slave-lhhq"
server_key: "8bedc6a54d47a77b57f0ffb967c79e9e"
server_key_iv: "8b9c9a52d6e2ea4a0d8fc6e3b2e79fe6"
server_key_version: "encryptionkey@0"
I20260926 21:40:03.578651 14219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f"
format_stamp: "Formatted at 2026-09-26 21:40:03 on dist-test-slave-lhhq"
server_key: "8bedc6a54d47a77b57f0ffb967c79e9e"
server_key_iv: "8b9c9a52d6e2ea4a0d8fc6e3b2e79fe6"
server_key_version: "encryptionkey@0"
I20260926 21:40:03.583870 14219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:40:03.587359 14234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:03.588764 14219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:40:03.588938 14219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f"
format_stamp: "Formatted at 2026-09-26 21:40:03 on dist-test-slave-lhhq"
server_key: "8bedc6a54d47a77b57f0ffb967c79e9e"
server_key_iv: "8b9c9a52d6e2ea4a0d8fc6e3b2e79fe6"
server_key_version: "encryptionkey@0"
I20260926 21:40:03.589133 14219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:03.606096 14219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:03.610720 14219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:03.611029 14219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:03.629410 14219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39563
I20260926 21:40:03.629426 14285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39563 every 8 connection(s)
I20260926 21:40:03.630836 14219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:40:03.634563 14286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:03.638785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 14219
I20260926 21:40:03.639008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:40:03.639660 388 external_mini_cluster.cc:1428] Setting key a1c7ec8f676d8d517ddad5934dedb4b4
I20260926 21:40:03.646930 14286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Bootstrap starting.
I20260926 21:40:03.650691 14286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:03.652029 14286 log.cc:824] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:03.655184 14286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: No bootstrap required, opened a new log
Sep 26 21:40:03 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:03.664963 14286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:03.665336 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:03.641763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.0.97.62:39563)
0926 21:40:03.642608 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:03.642629 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:03.642680 (+ 51us) server_negotiation.cc:407] Connection header received
0926 21:40:03.645261 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:03.645293 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:03.645806 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:03.646266 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:03.647657 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:03.649711 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:03.651192 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:03.652264 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:03.652478 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:03.657176 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:03.657256 (+ 80us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:03.657285 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:03.657336 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:03.660741 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:03.661725 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:03.661732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:03.661740 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:03.661947 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:03.662373 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:03.662376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:03.662378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:03.662983 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:03.663213 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:03.663735 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 21:40:03.664217 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":361,"threads_started":1}
I20260926 21:40:03.665446 14286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:03.665567 14286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9a1e52ef61483fa10e6bb8edbd3e4f, State: Initialized, Role: FOLLOWER
I20260926 21:40:03.666293 14286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:03.666555 14286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:40:03.666724 14286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:40:03.666942 14286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:03.669934 14286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:03.670401 14286 leader_election.cc:302] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a9a1e52ef61483fa10e6bb8edbd3e4f; no voters:
I20260926 21:40:03.670996 14286 leader_election.cc:288] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:40:03.671141 14291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:40:03.671928 14291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 1 LEADER]: Becoming Leader. State: Replica: 8a9a1e52ef61483fa10e6bb8edbd3e4f, State: Running, Role: LEADER
I20260926 21:40:03.672636 14291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:03.673429 14286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:40:03.675714 14292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } } }
I20260926 21:40:03.675690 14293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a9a1e52ef61483fa10e6bb8edbd3e4f. Latest consensus state: current_term: 1 leader_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } } }
I20260926 21:40:03.676146 14292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:03.676149 14293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:03.679637 14298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:40:03.686663 14298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:40:03.696010 14298 catalog_manager.cc:1347] Generated new cluster ID: 0e3437b1b9e44d72b0323855d717bdab
I20260926 21:40:03.696138 14298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:40:03.710126 14298 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:40:03.711499 14298 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:40:03.729387 14298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Generated new TSK 0
I20260926 21:40:03.730567 14298 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:40:03.797521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39563
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:04.127979 14312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:04.128265 14312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:04.128371 14312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:04.135564 14312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:04.135672 14312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:04.135807 14312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:40:04.145409 14312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:04.146610 14312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:04.147842 14312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:04.161217 14317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:04.161126 14320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:04.161161 14318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:04.162330 14312 server_base.cc:1034] running on GCE node
I20260926 21:40:04.162986 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:04.163880 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:04.165107 14312 hybrid_clock.cc:648] HybridClock initialized: now 1790458804165035 us; error 67 us; skew 500 ppm
I20260926 21:40:04.165449 14312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:04 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:04.169353 14312 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:40:04.170852 14312 webserver.cc:460] Webserver started at http://127.0.97.1:44721/ using document root <none> and password file <none>
I20260926 21:40:04.171556 14312 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:04.171662 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:04.171934 14312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:40:04.175346 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "9589bc1c14db43b5a1074273e1f3143f"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "137970db97df85336ae9209e023910f3"
server_key_iv: "5f36323910becd6e528998ffe413f74f"
server_key_version: "encryptionkey@0"
I20260926 21:40:04.176123 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9589bc1c14db43b5a1074273e1f3143f"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "137970db97df85336ae9209e023910f3"
server_key_iv: "5f36323910becd6e528998ffe413f74f"
server_key_version: "encryptionkey@0"
I20260926 21:40:04.181691 14312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 21:40:04.185261 14327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:04.186702 14312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:40:04.186899 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "9589bc1c14db43b5a1074273e1f3143f"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "137970db97df85336ae9209e023910f3"
server_key_iv: "5f36323910becd6e528998ffe413f74f"
server_key_version: "encryptionkey@0"
I20260926 21:40:04.187108 14312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:04.214928 14312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:04.219344 14312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:04.219650 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:04.220609 14312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:04.221973 14312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:04.222044 14312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:04.222189 14312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:04.222254 14312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:04.254961 14312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35239
I20260926 21:40:04.255002 14439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35239 every 8 connection(s)
I20260926 21:40:04.256265 14312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:40:04.261149 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 14312
I20260926 21:40:04.261441 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:40:04.262072 388 external_mini_cluster.cc:1428] Setting key 39535af1bdf5af1940c30ab428133ad9
Sep 26 21:40:04 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:04.277779 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:04.258998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34249 (local address 127.0.97.62:39563)
0926 21:40:04.259458 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:04.259463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:04.260294 (+ 831us) server_negotiation.cc:407] Connection header received
0926 21:40:04.262373 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:04.262380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:04.262473 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:04.262591 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:04.264335 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:04.265068 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:04.266670 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:04.267079 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:04.267220 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:04.271205 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:04.271214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:04.271221 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:04.271271 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:04.274066 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:04.275134 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:04.275138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:04.275140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:04.275262 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:04.275825 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:04.275828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:04.275830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:04.276088 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:04.276198 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:04.277278 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0926 21:40:04.277459 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":250,"threads_started":1}
I20260926 21:40:04.279389 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:04.259432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:34249)
0926 21:40:04.260133 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:04.260171 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:04.262060 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:04.262869 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:04.262881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:04.263488 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:04.264169 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:04.264188 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:04.265254 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:04.265266 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:04.266510 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:04.266522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:04.267313 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:04.267322 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:04.267574 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:04.268369 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:04.268395 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:04.270952 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:04.274262 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:04.274274 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:04.274296 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:04.275003 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:04.275389 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:04.275396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:04.275403 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:04.275693 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:04.276242 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:04.276254 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:04.276754 (+ 500us) client_negotiation.cc:769] Sending connection context
0926 21:40:04.277115 (+ 361us) client_negotiation.cc:240] Negotiation successful
0926 21:40:04.277774 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":281,"threads_started":1}
I20260926 21:40:04.281080 14440 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39563
I20260926 21:40:04.281442 14440 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:04.282222 14440 heartbeater.cc:511] Master 127.0.97.62:39563 requested a full tablet report, sending...
I20260926 21:40:04.284327 14251 ts_manager.cc:194] Registered new tserver with Master: 9589bc1c14db43b5a1074273e1f3143f (127.0.97.1:35239)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:40:04.285931 14251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:40:04.316766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39563
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:04.646759 14446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:04.647075 14446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:04.647212 14446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:04.655066 14446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:04.655197 14446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:04.655330 14446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:40:04.666366 14446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:04.667778 14446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:04.669055 14446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:04.683295 14452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:04.683327 14454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:04.683245 14451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:04.684232 14446 server_base.cc:1034] running on GCE node
I20260926 21:40:04.684887 14446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:04.685945 14446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:04.687178 14446 hybrid_clock.cc:648] HybridClock initialized: now 1790458804687123 us; error 54 us; skew 500 ppm
I20260926 21:40:04.687561 14446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:04 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:04.690923 14446 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:40:04.692458 14446 webserver.cc:460] Webserver started at http://127.0.97.2:39263/ using document root <none> and password file <none>
I20260926 21:40:04.693289 14446 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:04.693398 14446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:04.693688 14446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:40:04.696585 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "dfc18cce451547b489e8ca8a3da57761"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "60187e15ec515051951db07ae39b7d97"
server_key_iv: "0e85dedd08e6887d437ba409a53bd154"
server_key_version: "encryptionkey@0"
I20260926 21:40:04.697475 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dfc18cce451547b489e8ca8a3da57761"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "60187e15ec515051951db07ae39b7d97"
server_key_iv: "0e85dedd08e6887d437ba409a53bd154"
server_key_version: "encryptionkey@0"
I20260926 21:40:04.702678 14446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:40:04.706162 14461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:04.707530 14446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:40:04.707711 14446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "dfc18cce451547b489e8ca8a3da57761"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "60187e15ec515051951db07ae39b7d97"
server_key_iv: "0e85dedd08e6887d437ba409a53bd154"
server_key_version: "encryptionkey@0"
I20260926 21:40:04.707940 14446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:04.728849 14446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:04.733539 14446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:04.733863 14446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:04.734899 14446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:04.736331 14446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:04.736406 14446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:04.736572 14446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:04.736624 14446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:04.770989 14446 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44119
I20260926 21:40:04.771070 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44119 every 8 connection(s)
I20260926 21:40:04.772292 14446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:40:04.779860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 14446
I20260926 21:40:04.780059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:40:04.780493 388 external_mini_cluster.cc:1428] Setting key 4a32543fc67b7a7bbf379a50c9b157bd
Sep 26 21:40:04 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:04.795372 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:04.775124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33271 (local address 127.0.97.62:39563)
0926 21:40:04.775299 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:04.775315 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:04.776419 (+ 1104us) server_negotiation.cc:407] Connection header received
0926 21:40:04.778247 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:04.778251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:04.778322 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:04.778463 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:04.780178 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:04.780892 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:04.782940 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:04.783351 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:04.783462 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:04.788223 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:04.788232 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:04.788241 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:04.788294 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:04.791624 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:04.792731 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:04.792736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:04.792739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:04.792880 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:04.793452 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:04.793456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:04.793458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:04.793789 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:04.793912 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:04.794908 (+ 996us) server_negotiation.cc:299] Negotiation successful
0926 21:40:04.795125 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:40:04.796955 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:04.775522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:33271)
0926 21:40:04.776232 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:04.776272 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:04.777986 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:04.778679 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:04.778688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:04.779182 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:04.779961 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:04.779990 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:04.781091 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:04.781095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:04.782787 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:04.782802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:04.783587 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:04.783597 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:04.783939 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:04.784712 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:04.784752 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:04.787910 (+ 3158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:04.791802 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:04.791815 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:04.791839 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:04.792540 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:04.793001 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:04.793010 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:04.793016 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:04.793319 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:04.793931 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:04.793943 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:04.794462 (+ 519us) client_negotiation.cc:769] Sending connection context
0926 21:40:04.794784 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 21:40:04.795415 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":278,"threads_started":1}
I20260926 21:40:04.798682 14574 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39563
I20260926 21:40:04.799026 14574 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:04.799775 14574 heartbeater.cc:511] Master 127.0.97.62:39563 requested a full tablet report, sending...
I20260926 21:40:04.801442 14251 ts_manager.cc:194] Registered new tserver with Master: dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119)
I20260926 21:40:04.802400 14251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33271
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:40:04.837496 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39563
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:05.170008 14579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:05.170356 14579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:05.170518 14579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:05.178041 14579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:05.178190 14579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:05.178377 14579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:40:05.188186 14579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:05.189574 14579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:05.190860 14579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:05.204435 14584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:05.204423 14585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:05.204402 14587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:05.205062 14579 server_base.cc:1034] running on GCE node
I20260926 21:40:05.205701 14579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:05.206640 14579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:05.207878 14579 hybrid_clock.cc:648] HybridClock initialized: now 1790458805207827 us; error 58 us; skew 500 ppm
I20260926 21:40:05.208202 14579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:05 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:05.211663 14579 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:40:05.213274 14579 webserver.cc:460] Webserver started at http://127.0.97.3:40197/ using document root <none> and password file <none>
I20260926 21:40:05.213977 14579 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:05.214107 14579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:05.214426 14579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:40:05.217331 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "7e004c556894462883339583e520e373"
format_stamp: "Formatted at 2026-09-26 21:40:05 on dist-test-slave-lhhq"
server_key: "947c22fd8ba76957c39689a90d199b9f"
server_key_iv: "c3fc73d9ea962704aca6575139860ce8"
server_key_version: "encryptionkey@0"
I20260926 21:40:05.218122 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7e004c556894462883339583e520e373"
format_stamp: "Formatted at 2026-09-26 21:40:05 on dist-test-slave-lhhq"
server_key: "947c22fd8ba76957c39689a90d199b9f"
server_key_iv: "c3fc73d9ea962704aca6575139860ce8"
server_key_version: "encryptionkey@0"
I20260926 21:40:05.223241 14579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260926 21:40:05.226989 14594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:05.228340 14579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:40:05.228515 14579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "7e004c556894462883339583e520e373"
format_stamp: "Formatted at 2026-09-26 21:40:05 on dist-test-slave-lhhq"
server_key: "947c22fd8ba76957c39689a90d199b9f"
server_key_iv: "c3fc73d9ea962704aca6575139860ce8"
server_key_version: "encryptionkey@0"
I20260926 21:40:05.228717 14579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:05.262648 14579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:05.267266 14579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:05.267580 14579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:05.268601 14579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:05.270128 14579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:05.270231 14579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:05.270390 14579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:05.270463 14579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:05.289067 14440 heartbeater.cc:503] Master 127.0.97.62:39563 was elected leader, sending a full tablet report...
I20260926 21:40:05.305902 14579 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36053
I20260926 21:40:05.306046 14706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36053 every 8 connection(s)
I20260926 21:40:05.307235 14579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:40:05.310952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 14579
I20260926 21:40:05.311131 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:40:05.311726 388 external_mini_cluster.cc:1428] Setting key be5608d7a18d437de9bca3832733b1b5
Sep 26 21:40:05 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:05.328049 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.309950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44221 (local address 127.0.97.62:39563)
0926 21:40:05.310387 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.310392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.313797 (+ 3405us) server_negotiation.cc:407] Connection header received
0926 21:40:05.313856 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.313860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.313950 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.314065 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:05.315615 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.316523 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.318213 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.318651 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.318761 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.322238 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:05.322243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:05.322247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:05.322294 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:05.324789 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:05.325653 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:05.325657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:05.325659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:05.325836 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:05.326356 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:05.326362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:05.326365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:05.326682 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:05.326856 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.327593 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.327770 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":230,"threads_started":1}
I20260926 21:40:05.329010 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.310379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:44221)
0926 21:40:05.311219 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:05.311275 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:05.313471 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:05.314273 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:05.314281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:05.314887 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:05.315441 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.315457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.316698 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.316702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.318056 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.318068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.318849 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.318854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.319052 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.319772 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:05.319800 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:05.322029 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:05.324935 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:05.324942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:05.324955 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:05.325497 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:05.325968 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:05.325972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:05.325974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:05.326250 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:05.326827 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:05.326836 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:05.327193 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 21:40:05.327487 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 21:40:05.327906 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":332,"threads_started":1}
I20260926 21:40:05.330371 14707 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39563
I20260926 21:40:05.330675 14707 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:05.331414 14707 heartbeater.cc:511] Master 127.0.97.62:39563 requested a full tablet report, sending...
I20260926 21:40:05.332880 14251 ts_manager.cc:194] Registered new tserver with Master: 7e004c556894462883339583e520e373 (127.0.97.3:36053)
I20260926 21:40:05.333798 14251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44221
I20260926 21:40:05.339926 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:40:05.357324 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.344368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.0.97.62:39563)
0926 21:40:05.344544 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.344549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.345104 (+ 555us) server_negotiation.cc:407] Connection header received
0926 21:40:05.345301 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.345306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.345386 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.345487 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:05.347043 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.347810 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.349170 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.349637 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.349739 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.352150 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:05.352155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:05.352158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:05.352194 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:05.354763 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:05.355457 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:05.355461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:05.355463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:05.355579 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:05.355905 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:05.355908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:05.355910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:05.356222 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:05.356423 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.356735 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.356946 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:40:05.361824 14251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:40:05.369627 14251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:40:05.391660 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.382652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.0.97.1:35239)
0926 21:40:05.383420 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.383435 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.384537 (+ 1102us) server_negotiation.cc:407] Connection header received
0926 21:40:05.384645 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.384651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.384946 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.385138 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:05.386207 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.387389 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.389229 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.390451 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.390630 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.390870 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.391138 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.391299 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":301,"threads_started":1}
I20260926 21:40:05.391677 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.382692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:49640)
0926 21:40:05.384100 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:05.384147 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:05.384348 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:05.385463 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:05.385470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:05.385502 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:05.386012 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.386026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.387518 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.387526 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.389013 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.389028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.390673 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.390679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.390877 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.390945 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 21:40:05.391090 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 21:40:05.391362 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":936,"threads_started":1}
I20260926 21:40:05.395489 14375 tablet_service.cc:1467] Processing CreateTablet for tablet bb68fc67f0884d97a60cd54381313e50 (DEFAULT_TABLE table=test-table [id=313769fbecfc4e71b1f14ac86d9db86a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:05.396919 14375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb68fc67f0884d97a60cd54381313e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:05.406287 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.384577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.0.97.3:36053)
0926 21:40:05.397516 (+ 12939us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.397524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.397561 (+ 37us) server_negotiation.cc:407] Connection header received
0926 21:40:05.397664 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.397674 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.397924 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.398770 (+ 846us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:05.399074 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.400663 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.403568 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.405039 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.405210 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.405462 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.405740 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.405921 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12744,"thread_start_us":12582,"threads_started":1}
I20260926 21:40:05.406282 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.383820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36053 (local address 127.0.0.1:37570)
0926 21:40:05.384365 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:05.384381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:05.384494 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:05.398168 (+ 13674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:05.398176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:05.398206 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:05.398550 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.398563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.400831 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.400840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.403395 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.403412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.405307 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.405317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.405537 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.405577 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 21:40:05.405683 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 21:40:05.405862 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":286,"threads_started":1}
I20260926 21:40:05.406881 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.384565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44119 (local address 127.0.0.1:40078)
0926 21:40:05.385067 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:05.385085 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:05.385178 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:05.399536 (+ 14358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:05.399541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:05.399571 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:05.399959 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.399970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.401829 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.401835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.403988 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.403999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.405517 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.405524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.405704 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.405733 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 21:40:05.405794 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 21:40:05.405909 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":864,"threads_started":1}
I20260926 21:40:05.407998 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.385163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.0.97.2:44119)
0926 21:40:05.399038 (+ 13875us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.399045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.399070 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:40:05.399149 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.399154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.399332 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.399547 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:05.400144 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.401639 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.404164 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.405266 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.406062 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.406284 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.406377 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.407578 (+ 1201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13678,"thread_start_us":13528,"threads_started":1}
I20260926 21:40:05.410611 14642 tablet_service.cc:1467] Processing CreateTablet for tablet bb68fc67f0884d97a60cd54381313e50 (DEFAULT_TABLE table=test-table [id=313769fbecfc4e71b1f14ac86d9db86a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:05.411589 14509 tablet_service.cc:1467] Processing CreateTablet for tablet bb68fc67f0884d97a60cd54381313e50 (DEFAULT_TABLE table=test-table [id=313769fbecfc4e71b1f14ac86d9db86a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:05.412361 14642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb68fc67f0884d97a60cd54381313e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:05.412928 14726 tablet_bootstrap.cc:492] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: Bootstrap starting.
I20260926 21:40:05.413023 14509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb68fc67f0884d97a60cd54381313e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:05.417007 14726 tablet_bootstrap.cc:654] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:05.418216 14726 log.cc:824] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:05.421286 14726 tablet_bootstrap.cc:492] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: No bootstrap required, opened a new log
I20260926 21:40:05.421640 14726 ts_tablet_manager.cc:1397] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 21:40:05.427917 14728 tablet_bootstrap.cc:492] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: Bootstrap starting.
I20260926 21:40:05.429929 14729 tablet_bootstrap.cc:492] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: Bootstrap starting.
I20260926 21:40:05.432256 14728 tablet_bootstrap.cc:654] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:05.432075 14726 raft_consensus.cc:348] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } }
I20260926 21:40:05.432703 14726 raft_consensus.cc:374] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:05.432925 14726 raft_consensus.cc:729] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9589bc1c14db43b5a1074273e1f3143f, State: Initialized, Role: FOLLOWER
I20260926 21:40:05.433142 14729 tablet_bootstrap.cc:654] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:05.433851 14728 log.cc:824] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:05.433959 14726 consensus_queue.cc:260] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } }
I20260926 21:40:05.434470 14729 log.cc:824] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:05.438869 14726 ts_tablet_manager.cc:1428] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260926 21:40:05.440752 14729 tablet_bootstrap.cc:492] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: No bootstrap required, opened a new log
I20260926 21:40:05.441036 14729 ts_tablet_manager.cc:1397] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260926 21:40:05.441952 14728 tablet_bootstrap.cc:492] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: No bootstrap required, opened a new log
I20260926 21:40:05.442257 14728 ts_tablet_manager.cc:1397] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260926 21:40:05.449322 14729 raft_consensus.cc:348] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } }
I20260926 21:40:05.449736 14729 raft_consensus.cc:374] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:05.449841 14729 raft_consensus.cc:729] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc18cce451547b489e8ca8a3da57761, State: Initialized, Role: FOLLOWER
I20260926 21:40:05.450423 14729 consensus_queue.cc:260] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } }
I20260926 21:40:05.451594 14574 heartbeater.cc:503] Master 127.0.97.62:39563 was elected leader, sending a full tablet report...
I20260926 21:40:05.451730 14728 raft_consensus.cc:348] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } }
I20260926 21:40:05.452147 14728 raft_consensus.cc:374] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:05.452200 14729 ts_tablet_manager.cc:1428] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 21:40:05.452278 14728 raft_consensus.cc:729] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e004c556894462883339583e520e373, State: Initialized, Role: FOLLOWER
I20260926 21:40:05.452932 14728 consensus_queue.cc:260] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } }
I20260926 21:40:05.454072 14707 heartbeater.cc:503] Master 127.0.97.62:39563 was elected leader, sending a full tablet report...
I20260926 21:40:05.454960 14728 ts_tablet_manager.cc:1428] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
W20260926 21:40:05.508948 14441 tablet.cc:2369] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:40:05.524511 14575 tablet.cc:2369] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:40:05.559012 14708 tablet.cc:2369] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:40:05.648886 14734 raft_consensus.cc:482] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:40:05.649246 14734 raft_consensus.cc:504] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } }
I20260926 21:40:05.650921 14734 leader_election.cc:288] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9589bc1c14db43b5a1074273e1f3143f (127.0.97.1:35239), dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119)
I20260926 21:40:05.658263 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.651373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58607 (local address 127.0.97.1:35239)
0926 21:40:05.651620 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.651626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.651650 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:40:05.652099 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.652108 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.652197 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.652306 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:05.652770 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.653841 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.656255 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.657449 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.657591 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.657705 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.657917 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.658069 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:40:05.658370 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.651259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.97.3:58607)
0926 21:40:05.651514 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:05.651533 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:05.651676 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:05.652355 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:05.652360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:05.652390 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:05.652632 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.652643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.653978 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.653981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.656085 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.656102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.657614 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.657618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.657759 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.657784 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 21:40:05.657863 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 21:40:05.658034 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":32}
I20260926 21:40:05.658880 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.651711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44119 (local address 127.0.97.3:33621)
0926 21:40:05.652411 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:05.652429 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:05.652555 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:05.652912 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:05.652915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:05.652935 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:05.653241 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.653248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.654398 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.654404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.656428 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:05.656438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.658051 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.658057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:05.658230 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.658259 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 21:40:05.658360 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 21:40:05.658491 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":1,"thread_start_us":393,"threads_started":1}
I20260926 21:40:05.659116 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.651898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33621 (local address 127.0.97.2:44119)
0926 21:40:05.652159 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.652167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.652523 (+ 356us) server_negotiation.cc:407] Connection header received
0926 21:40:05.652672 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.652675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.652765 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.652859 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:05.653380 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.654252 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.656573 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.657818 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.657969 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.658056 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.658726 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.658863 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 21:40:05.659534 14395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb68fc67f0884d97a60cd54381313e50" candidate_uuid: "7e004c556894462883339583e520e373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9589bc1c14db43b5a1074273e1f3143f" is_pre_election: true
I20260926 21:40:05.660064 14395 raft_consensus.cc:2454] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e004c556894462883339583e520e373 in term 0.
I20260926 21:40:05.660086 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb68fc67f0884d97a60cd54381313e50" candidate_uuid: "7e004c556894462883339583e520e373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc18cce451547b489e8ca8a3da57761" is_pre_election: true
I20260926 21:40:05.660627 14529 raft_consensus.cc:2454] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e004c556894462883339583e520e373 in term 0.
I20260926 21:40:05.660802 14596 leader_election.cc:302] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e004c556894462883339583e520e373, 9589bc1c14db43b5a1074273e1f3143f; no voters:
I20260926 21:40:05.661230 14734 raft_consensus.cc:2781] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:40:05.661376 14734 raft_consensus.cc:482] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:40:05.661513 14734 raft_consensus.cc:3037] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:05.663715 14734 raft_consensus.cc:504] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } }
I20260926 21:40:05.664291 14734 leader_election.cc:288] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [CANDIDATE]: Term 1 election: Requested vote from peers 9589bc1c14db43b5a1074273e1f3143f (127.0.97.1:35239), dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119)
I20260926 21:40:05.664773 14395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb68fc67f0884d97a60cd54381313e50" candidate_uuid: "7e004c556894462883339583e520e373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9589bc1c14db43b5a1074273e1f3143f"
I20260926 21:40:05.664817 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb68fc67f0884d97a60cd54381313e50" candidate_uuid: "7e004c556894462883339583e520e373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc18cce451547b489e8ca8a3da57761"
I20260926 21:40:05.664973 14395 raft_consensus.cc:3037] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:05.665035 14529 raft_consensus.cc:3037] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:05.667196 14395 raft_consensus.cc:2454] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e004c556894462883339583e520e373 in term 1.
I20260926 21:40:05.667200 14529 raft_consensus.cc:2454] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e004c556894462883339583e520e373 in term 1.
I20260926 21:40:05.667605 14596 leader_election.cc:302] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e004c556894462883339583e520e373, 9589bc1c14db43b5a1074273e1f3143f; no voters:
I20260926 21:40:05.667960 14734 raft_consensus.cc:2781] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:40:05.668403 14734 raft_consensus.cc:686] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 1 LEADER]: Becoming Leader. State: Replica: 7e004c556894462883339583e520e373, State: Running, Role: LEADER
I20260926 21:40:05.668931 14734 consensus_queue.cc:237] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } }
I20260926 21:40:05.673629 14250 catalog_manager.cc:5495] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e004c556894462883339583e520e373 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7e004c556894462883339583e520e373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } health_report { overall_health: HEALTHY } } }
I20260926 21:40:05.691524 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.686024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.0.97.3:36053)
0926 21:40:05.686229 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.686233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.686251 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:40:05.686409 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.686413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.686489 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.686613 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:40:05.686972 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.687677 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.689103 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.689633 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.689764 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.690229 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:40:05.690886 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:40:05.691008 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.691101 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.691173 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:40:05.699990 14529 raft_consensus.cc:1261] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [term 1 FOLLOWER]: Refusing update from remote peer 7e004c556894462883339583e520e373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:40:05.699990 14395 raft_consensus.cc:1261] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Refusing update from remote peer 7e004c556894462883339583e520e373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:40:05.700953 14736 consensus_queue.cc:1035] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:40:05.701422 14734 consensus_queue.cc:1035] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:40:05.729661 14250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:40:05.730019 14250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:40:05.734103 14250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Sending DeleteTablet for 3 replicas of tablet bb68fc67f0884d97a60cd54381313e50
I20260926 21:40:05.735389 14375 tablet_service.cc:1514] Processing DeleteTablet for tablet bb68fc67f0884d97a60cd54381313e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49640
I20260926 21:40:05.735363 14642 tablet_service.cc:1514] Processing DeleteTablet for tablet bb68fc67f0884d97a60cd54381313e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37570
I20260926 21:40:05.735697 14509 tablet_service.cc:1514] Processing DeleteTablet for tablet bb68fc67f0884d97a60cd54381313e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40078
I20260926 21:40:05.737033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 14312
I20260926 21:40:05.748914 14750 tablet_replica.cc:330] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: stopping tablet replica
I20260926 21:40:05.749773 14750 raft_consensus.cc:2229] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:40:05.750639 14750 raft_consensus.cc:2258] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:05.751653 14749 tablet_replica.cc:330] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: stopping tablet replica
I20260926 21:40:05.752066 14749 raft_consensus.cc:2229] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:40:05.752655 14749 raft_consensus.cc:2258] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:05.754828 14749 ts_tablet_manager.cc:1905] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:05.754828 14750 ts_tablet_manager.cc:1905] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:05.755198 14751 tablet_replica.cc:330] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: stopping tablet replica
I20260926 21:40:05.755641 14751 raft_consensus.cc:2229] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:40:05.756150 14751 raft_consensus.cc:2258] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:05.762504 14750 ts_tablet_manager.cc:1918] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:05.762694 14750 log.cc:1196] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/bb68fc67f0884d97a60cd54381313e50
I20260926 21:40:05.762952 14751 ts_tablet_manager.cc:1905] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:05.763252 14750 ts_tablet_manager.cc:1939] T bb68fc67f0884d97a60cd54381313e50 P 9589bc1c14db43b5a1074273e1f3143f: Deleting consensus metadata
I20260926 21:40:05.763458 14749 ts_tablet_manager.cc:1918] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:05.763684 14749 log.cc:1196] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/bb68fc67f0884d97a60cd54381313e50
I20260926 21:40:05.764271 14749 ts_tablet_manager.cc:1939] T bb68fc67f0884d97a60cd54381313e50 P 7e004c556894462883339583e520e373: Deleting consensus metadata
I20260926 21:40:05.765344 14236 catalog_manager.cc:4841] TS 9589bc1c14db43b5a1074273e1f3143f (127.0.97.1:35239): tablet bb68fc67f0884d97a60cd54381313e50 (table test-table [id=313769fbecfc4e71b1f14ac86d9db86a]) successfully deleted
I20260926 21:40:05.766085 14237 catalog_manager.cc:4841] TS 7e004c556894462883339583e520e373 (127.0.97.3:36053): tablet bb68fc67f0884d97a60cd54381313e50 (table test-table [id=313769fbecfc4e71b1f14ac86d9db86a]) successfully deleted
I20260926 21:40:05.770212 14751 ts_tablet_manager.cc:1918] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:05.770411 14751 log.cc:1196] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/bb68fc67f0884d97a60cd54381313e50
I20260926 21:40:05.771087 14751 ts_tablet_manager.cc:1939] T bb68fc67f0884d97a60cd54381313e50 P dfc18cce451547b489e8ca8a3da57761: Deleting consensus metadata
I20260926 21:40:05.773074 14236 catalog_manager.cc:4841] TS dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119): tablet bb68fc67f0884d97a60cd54381313e50 (table test-table [id=313769fbecfc4e71b1f14ac86d9db86a]) successfully deleted
Sep 26 21:40:05 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:40:05.785808 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.738205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.0.97.1:35239)
0926 21:40:05.738391 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.738398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.745608 (+ 7210us) server_negotiation.cc:407] Connection header received
0926 21:40:05.745860 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.745866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.745958 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.746079 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:05.756675 (+ 10596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.757884 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.760338 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.761047 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.761239 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.778163 (+ 16924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:05.778185 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:05.778211 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:05.778278 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:05.782290 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:05.783360 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:05.783371 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:05.783375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:05.783539 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:05.784006 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:05.784011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:05.784014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:05.784480 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:05.784808 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.785137 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.785422 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 21:40:05.786563 14435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:05.906391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 14312
I20260926 21:40:05.924178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 14446
Sep 26 21:40:05 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:40:05.941620 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:05.925250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.0.97.2:44119)
0926 21:40:05.925470 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:05.925476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:05.925499 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:40:05.925569 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:05.925574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:05.925671 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:05.925780 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:05.927018 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.928164 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.929521 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:05.930184 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:05.930332 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:05.934804 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:05.934823 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:05.934845 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:05.934896 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:05.938645 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:05.939362 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:05.939372 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:05.939376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:05.939544 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:05.939893 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:05.939899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:05.939902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:05.940387 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:05.940683 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:05.940923 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 21:40:05.941175 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:40:05.942350 14569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:06.033141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 14446
I20260926 21:40:06.052392 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 14579
Sep 26 21:40:06 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:40:06.070608 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:06.053348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.0.97.3:36053)
0926 21:40:06.053525 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:06.053530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:06.053548 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:40:06.053687 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:06.053694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:06.053755 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:06.053887 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:06.055747 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:06.056573 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:06.058528 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:06.059008 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:06.059113 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:06.064159 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:06.064174 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:06.064189 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:06.064229 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:06.067530 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:06.068286 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:06.068294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:06.068296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:06.068456 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:06.068859 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:06.068868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:06.068870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:06.069292 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:06.069644 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:06.069971 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 21:40:06.070201 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:40:06.071465 14702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:06.195549 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 14579
I20260926 21:40:06.215827 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 14219
I20260926 21:40:06.217012 14281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:06.327291 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 14219
I20260926 21:40:06.348505 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39563
--webserver_interface=127.0.97.62
--webserver_port=37651
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:06.681994 14757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:06.682289 14757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:06.682384 14757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:06.689783 14757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:40:06.689898 14757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:06.689953 14757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:06.690009 14757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:40:06.690057 14757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:40:06.699855 14757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:06.701093 14757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:06.702394 14757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:06.724826 14764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:06.724781 14766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:06.724748 14763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:06.725890 14757 server_base.cc:1034] running on GCE node
I20260926 21:40:06.726842 14757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:06.728464 14757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:06.729744 14757 hybrid_clock.cc:648] HybridClock initialized: now 1790458806729694 us; error 55 us; skew 500 ppm
I20260926 21:40:06.730235 14757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:06 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:06.734165 14757 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:40:06.736048 14757 webserver.cc:460] Webserver started at http://127.0.97.62:37651/ using document root <none> and password file <none>
I20260926 21:40:06.736881 14757 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:06.737056 14757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:06.743357 14757 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:40:06.746400 14773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:06.748013 14757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:40:06.748214 14757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f"
format_stamp: "Formatted at 2026-09-26 21:40:03 on dist-test-slave-lhhq"
server_key: "8bedc6a54d47a77b57f0ffb967c79e9e"
server_key_iv: "8b9c9a52d6e2ea4a0d8fc6e3b2e79fe6"
server_key_version: "encryptionkey@0"
I20260926 21:40:06.749425 14757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:06.776019 14757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:06.780524 14757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:06.780851 14757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:06.801020 14757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39563
I20260926 21:40:06.801029 14824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39563 every 8 connection(s)
I20260926 21:40:06.802765 14757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:40:06.810135 14825 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:40:06.813148 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 14757
I20260926 21:40:06.813745 14825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Bootstrap starting.
I20260926 21:40:06.814070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35239
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39563
--webserver_port=44721
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:40:06.833909 14825 log.cc:824] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:06.846483 14825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:40:06.847059 14825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Bootstrap complete.
I20260926 21:40:06.856567 14825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:06.857527 14825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9a1e52ef61483fa10e6bb8edbd3e4f, State: Initialized, Role: FOLLOWER
I20260926 21:40:06.858157 14825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:06.858390 14825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:40:06.858505 14825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:40:06.858682 14825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:40:06.862774 14825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:06.863304 14825 leader_election.cc:302] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a9a1e52ef61483fa10e6bb8edbd3e4f; no voters:
I20260926 21:40:06.863942 14825 leader_election.cc:288] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 21:40:06.864089 14829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:40:06.866544 14825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:40:06.871253 14829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 2 LEADER]: Becoming Leader. State: Replica: 8a9a1e52ef61483fa10e6bb8edbd3e4f, State: Running, Role: LEADER
I20260926 21:40:06.871999 14829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:06.876886 14831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a9a1e52ef61483fa10e6bb8edbd3e4f. Latest consensus state: current_term: 2 leader_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } } }
I20260926 21:40:06.877357 14831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:06.886212 14830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } } }
I20260926 21:40:06.886631 14830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:06.887565 14838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:40:06.896781 14838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:40:06.898530 14838 catalog_manager.cc:1259] Loaded cluster ID: 0e3437b1b9e44d72b0323855d717bdab
I20260926 21:40:06.898690 14838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:40:06.902206 14838 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 21:40:06.920717 14838 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 21:40:06.929979 14838 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:40:06.931676 14838 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Loaded TSK: 0
I20260926 21:40:06.933238 14838 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 21:40:07.170831 14827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:07.171156 14827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:07.171329 14827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:07.178617 14827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:07.178777 14827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:07.178974 14827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:40:07.189118 14827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44721
--tserver_master_addrs=127.0.97.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:07.190472 14827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:07.191835 14827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:07.205729 14855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:07.205687 14853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:07.206693 14827 server_base.cc:1034] running on GCE node
W20260926 21:40:07.209486 14852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:07.210240 14827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:07.211324 14827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:07.212550 14827 hybrid_clock.cc:648] HybridClock initialized: now 1790458807212508 us; error 40 us; skew 500 ppm
I20260926 21:40:07.212890 14827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:07 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:07.228845 14827 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:40:07.230655 14827 webserver.cc:460] Webserver started at http://127.0.97.1:44721/ using document root <none> and password file <none>
I20260926 21:40:07.231411 14827 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:07.231560 14827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:07.237677 14827 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 21:40:07.240593 14862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:07.242223 14827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:40:07.242436 14827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "9589bc1c14db43b5a1074273e1f3143f"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "137970db97df85336ae9209e023910f3"
server_key_iv: "5f36323910becd6e528998ffe413f74f"
server_key_version: "encryptionkey@0"
I20260926 21:40:07.243490 14827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:07.262017 14827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:07.266512 14827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:07.266809 14827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:07.267906 14827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:07.269390 14827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:07.269511 14827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:07.269647 14827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:07.269742 14827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:07.303699 14827 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35239
I20260926 21:40:07.303768 14974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35239 every 8 connection(s)
I20260926 21:40:07.305177 14827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:40:07.310343 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 14827
I20260926 21:40:07.310994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44119
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39563
--webserver_port=39263
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:40:07 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:07.330698 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:07.308374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:48907)
0926 21:40:07.309296 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:07.309352 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:07.311267 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:07.313002 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:07.313010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:07.313606 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:07.314140 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:07.314157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:07.316583 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:07.316587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:07.317887 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:07.317899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:07.318936 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:07.318943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:07.319249 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:07.319992 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:07.320026 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:07.322667 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:07.326171 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:07.326181 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:07.326197 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:07.326634 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:07.327098 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:07.327103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:07.327107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:07.327304 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:07.328141 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:07.328150 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:07.328512 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 21:40:07.328767 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 21:40:07.329343 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":391,"threads_started":1}
I20260926 21:40:07.330698 14977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:07.308083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48907 (local address 127.0.97.62:39563)
0926 21:40:07.308645 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:07.308656 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:07.309480 (+ 824us) server_negotiation.cc:407] Connection header received
0926 21:40:07.311921 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:07.311943 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:07.312452 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:07.312880 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:07.314382 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:07.316425 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:07.318063 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:07.318689 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:07.318821 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:07.322946 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:07.322954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:07.322975 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:07.323011 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:07.326003 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:07.326789 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:07.326796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:07.326802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:07.326979 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:07.327430 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:07.327435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:07.327437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:07.327998 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:07.328215 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:07.329003 (+ 788us) server_negotiation.cc:299] Negotiation successful
0926 21:40:07.329413 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260926 21:40:07.333804 14975 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39563
I20260926 21:40:07.334123 14975 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:07.334820 14975 heartbeater.cc:511] Master 127.0.97.62:39563 requested a full tablet report, sending...
I20260926 21:40:07.337004 14790 ts_manager.cc:194] Registered new tserver with Master: 9589bc1c14db43b5a1074273e1f3143f (127.0.97.1:35239)
I20260926 21:40:07.338527 14790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48907
W20260926 21:40:07.644158 14979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:07.644595 14979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:07.644783 14979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:07.652345 14979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:07.652514 14979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:07.652693 14979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:40:07.662876 14979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39263
--tserver_master_addrs=127.0.97.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:07.664724 14979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:07.666231 14979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:07.680285 14984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:07.680328 14987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:07.680305 14985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:07.681023 14979 server_base.cc:1034] running on GCE node
I20260926 21:40:07.681739 14979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:07.682710 14979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:07.684029 14979 hybrid_clock.cc:648] HybridClock initialized: now 1790458807683950 us; error 58 us; skew 500 ppm
I20260926 21:40:07.684437 14979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:07 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:07.687961 14979 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:40:07.689687 14979 webserver.cc:460] Webserver started at http://127.0.97.2:39263/ using document root <none> and password file <none>
I20260926 21:40:07.690444 14979 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:07.690618 14979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:07.696501 14979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 21:40:07.699391 14994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:07.700865 14979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:40:07.701056 14979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "dfc18cce451547b489e8ca8a3da57761"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "60187e15ec515051951db07ae39b7d97"
server_key_iv: "0e85dedd08e6887d437ba409a53bd154"
server_key_version: "encryptionkey@0"
I20260926 21:40:07.702199 14979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:07.724113 14979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:07.730430 14979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:07.730749 14979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:07.731918 14979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:07.733474 14979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:07.733603 14979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:07.733753 14979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:07.733861 14979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:07.771224 14979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44119
I20260926 21:40:07.771286 15106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44119 every 8 connection(s)
I20260926 21:40:07.772848 14979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:40:07.776060 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 14979
I20260926 21:40:07.776775 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36053
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39563
--webserver_port=40197
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:40:07 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:07.794621 14977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:07.775714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38301 (local address 127.0.97.62:39563)
0926 21:40:07.776970 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:07.776977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:07.777580 (+ 603us) server_negotiation.cc:407] Connection header received
0926 21:40:07.779569 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:07.779575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:07.779678 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:07.779795 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:07.781466 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:07.782185 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:07.783797 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:07.784241 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:07.784380 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:07.788302 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:07.788307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:07.788310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:07.788389 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:07.791207 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:07.792110 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:07.792121 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:07.792123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:07.792258 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:07.792733 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:07.792737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:07.792739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:07.793026 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:07.793192 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:07.793999 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 21:40:07.794253 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094}
I20260926 21:40:07.795696 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:07.776414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:38301)
0926 21:40:07.777411 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:07.777458 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:07.779256 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:07.780069 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:07.780078 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:07.780626 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:07.781280 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:07.781301 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:07.782368 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:07.782373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:07.783644 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:07.783656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:07.784492 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:07.784499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:07.784729 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:07.785508 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:07.785536 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:07.788068 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:07.791407 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:07.791418 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:07.791442 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:07.791962 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:07.792378 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:07.792384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:07.792388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:07.792596 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:07.793162 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:07.793171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:07.793596 (+ 425us) client_negotiation.cc:769] Sending connection context
0926 21:40:07.793881 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 21:40:07.794482 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":428,"threads_started":1}
I20260926 21:40:07.797299 15107 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39563
I20260926 21:40:07.797626 15107 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:07.798375 15107 heartbeater.cc:511] Master 127.0.97.62:39563 requested a full tablet report, sending...
I20260926 21:40:07.799862 14790 ts_manager.cc:194] Registered new tserver with Master: dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119)
I20260926 21:40:07.801054 14790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38301
W20260926 21:40:08.113875 15110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:08.114173 15110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:08.114338 15110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:08.122108 15110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:08.122218 15110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:08.122417 15110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:40:08.132306 15110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40197
--tserver_master_addrs=127.0.97.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:08.133714 15110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:08.135051 15110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:08.148537 15116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:08.148551 15118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:08.148703 15115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:08.149485 15110 server_base.cc:1034] running on GCE node
I20260926 21:40:08.150058 15110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:08.151072 15110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:08.152285 15110 hybrid_clock.cc:648] HybridClock initialized: now 1790458808152242 us; error 40 us; skew 500 ppm
I20260926 21:40:08.152634 15110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:08 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:08.155843 15110 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:40:08.157377 15110 webserver.cc:460] Webserver started at http://127.0.97.3:40197/ using document root <none> and password file <none>
I20260926 21:40:08.158031 15110 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:08.158143 15110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:08.163455 15110 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 21:40:08.166152 15125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:08.167619 15110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:40:08.167815 15110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "7e004c556894462883339583e520e373"
format_stamp: "Formatted at 2026-09-26 21:40:05 on dist-test-slave-lhhq"
server_key: "947c22fd8ba76957c39689a90d199b9f"
server_key_iv: "c3fc73d9ea962704aca6575139860ce8"
server_key_version: "encryptionkey@0"
I20260926 21:40:08.168769 15110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:08.189056 15110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:08.193156 15110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:08.193454 15110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:08.194408 15110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:08.195783 15110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:08.195878 15110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:08.196045 15110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:08.196138 15110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:08.227944 15110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36053
I20260926 21:40:08.228035 15237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36053 every 8 connection(s)
I20260926 21:40:08.229444 15110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:40:08.231050 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 15110
Sep 26 21:40:08 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:08.252666 14977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.232108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.0.97.62:39563)
0926 21:40:08.232298 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.232307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.237844 (+ 5537us) server_negotiation.cc:407] Connection header received
0926 21:40:08.238084 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.238089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.238193 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.238312 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:08.239927 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.240758 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.242335 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.242948 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.243052 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.246154 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:08.246160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:08.246165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:08.246206 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:08.249365 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:08.250235 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:08.250240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:08.250242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:08.250404 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:08.250908 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:08.250913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:08.250916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:08.251348 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:08.251631 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.251936 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.252279 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":28800}
I20260926 21:40:08.254791 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.232115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44797 (local address 127.0.97.62:39563)
0926 21:40:08.238020 (+ 5905us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.238026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.238045 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:40:08.238101 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.238105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.238189 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.238309 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:08.239940 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.240762 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.242653 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.243247 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.243366 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.247305 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:08.247310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:08.247313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:08.247344 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:08.250698 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:08.251763 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:08.251769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:08.251774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:08.251918 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:08.252515 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:08.252519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:08.252521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:08.252812 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:08.252957 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.254114 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.254306 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":5772,"spinlock_wait_cycles":12928,"thread_start_us":5606,"threads_started":1}
I20260926 21:40:08.255820 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.232830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:44797)
0926 21:40:08.233976 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.234038 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:08.237491 (+ 3453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:08.238605 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:08.238614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:08.239147 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:08.239729 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.239746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.240925 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.240930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.242507 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.242520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.243462 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.243467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.243675 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.244218 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:08.244244 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:08.246979 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:08.250915 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:08.250925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:08.250952 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:08.251621 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:08.252135 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:08.252139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:08.252142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:08.252393 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:08.253032 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:08.253040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:08.253586 (+ 546us) client_negotiation.cc:769] Sending connection context
0926 21:40:08.253920 (+ 334us) client_negotiation.cc:240] Negotiation successful
0926 21:40:08.254498 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":438,"threads_started":1}
I20260926 21:40:08.257153 15238 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39563
I20260926 21:40:08.257530 15238 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:08.258255 15238 heartbeater.cc:511] Master 127.0.97.62:39563 requested a full tablet report, sending...
I20260926 21:40:08.259639 14790 ts_manager.cc:194] Registered new tserver with Master: 7e004c556894462883339583e520e373 (127.0.97.3:36053)
I20260926 21:40:08.260453 14790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44797
I20260926 21:40:08.268039 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:40:08.274824 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.269126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.0.97.62:39563)
0926 21:40:08.269395 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.269400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.269569 (+ 169us) server_negotiation.cc:407] Connection header received
0926 21:40:08.269749 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.269753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.269887 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.270018 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:40:08.270377 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.271092 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.272594 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.273021 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.273175 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.273726 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:40:08.274251 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:40:08.274364 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.274473 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.274541 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 21:40:08.275753 14790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:40:08.283715 14790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:40:08.319561 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.296526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.0.97.3:36053)
0926 21:40:08.309261 (+ 12735us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.309274 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.310196 (+ 922us) server_negotiation.cc:407] Connection header received
0926 21:40:08.310311 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.310319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.310555 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.310745 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:08.311658 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.313289 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.316828 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.318312 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.318451 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.318686 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.319036 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.319218 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12455,"thread_start_us":13465,"threads_started":1}
I20260926 21:40:08.320276 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.296438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36053 (local address 127.0.0.1:37590)
0926 21:40:08.309763 (+ 13325us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.309781 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:08.309909 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:08.311035 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:08.311042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:08.311073 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:08.311452 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.311465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.313517 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.313522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.316607 (+ 3085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.316626 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.318540 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.318546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.318744 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.318791 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 21:40:08.319832 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0926 21:40:08.319997 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13210,"thread_start_us":13080,"threads_started":1}
I20260926 21:40:08.322165 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.296831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.0.97.2:44119)
0926 21:40:08.310787 (+ 13956us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.310799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.310828 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:40:08.310947 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.310959 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.311239 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.311451 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:08.312443 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.316021 (+ 3578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.318631 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.320854 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.320985 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.321196 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.321815 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.321956 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13782,"thread_start_us":16403,"threads_started":1}
I20260926 21:40:08.322921 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.296395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44119 (local address 127.0.0.1:40086)
0926 21:40:08.309481 (+ 13086us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.309529 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:08.309716 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:08.311763 (+ 2047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:08.311771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:08.311807 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:08.312229 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.312244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.316265 (+ 4021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.316273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.318401 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.318421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.321080 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.321087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.321357 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.321385 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 21:40:08.321559 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 21:40:08.322524 (+ 965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12803,"mutex_wait_us":2,"thread_start_us":13511,"threads_started":1}
I20260926 21:40:08.324589 15173 tablet_service.cc:1467] Processing CreateTablet for tablet e3b14c8415dc4eec9b1637ee84686854 (DEFAULT_TABLE table=test-table [id=2a99b994461549329b740ead9e87fb6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:08.325819 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.313618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:49666)
0926 21:40:08.317306 (+ 3688us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.317324 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:08.317461 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:08.318060 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:08.318065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:08.318089 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:08.318523 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.318535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.320355 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.320363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.323214 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.323230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.325003 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.325009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.325177 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.325236 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 21:40:08.325330 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 21:40:08.325478 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3531,"mutex_wait_us":668,"thread_start_us":3415,"threads_started":1}
I20260926 21:40:08.326159 15173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b14c8415dc4eec9b1637ee84686854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:08.326578 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.310646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.0.97.1:35239)
0926 21:40:08.317015 (+ 6369us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.317024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.317435 (+ 411us) server_negotiation.cc:407] Connection header received
0926 21:40:08.317614 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.317622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.317843 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.318048 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:08.318726 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.320221 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.323377 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.324770 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.324904 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.325110 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.326084 (+ 974us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.326253 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6178,"thread_start_us":6028,"threads_started":1}
I20260926 21:40:08.326149 15042 tablet_service.cc:1467] Processing CreateTablet for tablet e3b14c8415dc4eec9b1637ee84686854 (DEFAULT_TABLE table=test-table [id=2a99b994461549329b740ead9e87fb6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:08.327435 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b14c8415dc4eec9b1637ee84686854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:08.329932 14910 tablet_service.cc:1467] Processing CreateTablet for tablet e3b14c8415dc4eec9b1637ee84686854 (DEFAULT_TABLE table=test-table [id=2a99b994461549329b740ead9e87fb6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:08.331254 14910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b14c8415dc4eec9b1637ee84686854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:08.339123 15254 tablet_bootstrap.cc:492] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: Bootstrap starting.
I20260926 21:40:08.341281 14975 heartbeater.cc:503] Master 127.0.97.62:39563 was elected leader, sending a full tablet report...
I20260926 21:40:08.343152 15254 tablet_bootstrap.cc:654] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:08.344892 15254 log.cc:824] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:08.346535 15255 tablet_bootstrap.cc:492] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: Bootstrap starting.
I20260926 21:40:08.348739 15254 tablet_bootstrap.cc:492] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: No bootstrap required, opened a new log
I20260926 21:40:08.349164 15254 ts_tablet_manager.cc:1397] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 21:40:08.349496 15257 tablet_bootstrap.cc:492] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Bootstrap starting.
I20260926 21:40:08.350579 15255 tablet_bootstrap.cc:654] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:08.351995 15255 log.cc:824] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:08.353044 15257 tablet_bootstrap.cc:654] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:08.354385 15257 log.cc:824] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:08.355885 15255 tablet_bootstrap.cc:492] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: No bootstrap required, opened a new log
I20260926 21:40:08.356240 15255 ts_tablet_manager.cc:1397] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 21:40:08.358608 15257 tablet_bootstrap.cc:492] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: No bootstrap required, opened a new log
I20260926 21:40:08.358911 15257 ts_tablet_manager.cc:1397] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 21:40:08.360500 15254 raft_consensus.cc:348] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:08.360999 15254 raft_consensus.cc:374] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:08.361145 15254 raft_consensus.cc:729] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc18cce451547b489e8ca8a3da57761, State: Initialized, Role: FOLLOWER
I20260926 21:40:08.361886 15254 consensus_queue.cc:260] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:08.363094 15107 heartbeater.cc:503] Master 127.0.97.62:39563 was elected leader, sending a full tablet report...
I20260926 21:40:08.363770 15254 ts_tablet_manager.cc:1428] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260926 21:40:08.367136 15255 raft_consensus.cc:348] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:08.367664 15255 raft_consensus.cc:374] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:08.367432 15257 raft_consensus.cc:348] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:08.367796 15255 raft_consensus.cc:729] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e004c556894462883339583e520e373, State: Initialized, Role: FOLLOWER
I20260926 21:40:08.367910 15257 raft_consensus.cc:374] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:08.368054 15257 raft_consensus.cc:729] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9589bc1c14db43b5a1074273e1f3143f, State: Initialized, Role: FOLLOWER
I20260926 21:40:08.368494 15255 consensus_queue.cc:260] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:08.368844 15257 consensus_queue.cc:260] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:08.369724 15238 heartbeater.cc:503] Master 127.0.97.62:39563 was elected leader, sending a full tablet report...
I20260926 21:40:08.370471 15255 ts_tablet_manager.cc:1428] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 21:40:08.372066 15257 ts_tablet_manager.cc:1428] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
W20260926 21:40:08.481248 15239 tablet.cc:2369] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:40:08.525131 15108 tablet.cc:2369] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:40:08.525899 15260 raft_consensus.cc:482] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:40:08.526217 15260 raft_consensus.cc:504] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:08.527935 15260 leader_election.cc:288] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e004c556894462883339583e520e373 (127.0.97.3:36053), 9589bc1c14db43b5a1074273e1f3143f (127.0.97.1:35239)
I20260926 21:40:08.536768 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.528417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36567 (local address 127.0.97.1:35239)
0926 21:40:08.528612 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.528619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.530320 (+ 1701us) server_negotiation.cc:407] Connection header received
0926 21:40:08.530667 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.530675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.530752 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.530833 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:08.531919 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.533007 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.535040 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.535804 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.535973 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.536087 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.536366 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.536494 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:40:08.536926 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.528444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.97.2:36567)
0926 21:40:08.530198 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.530219 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:08.530443 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:08.530948 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:08.530953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:08.530988 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:08.531627 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.531643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.533154 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.533157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.534869 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.534881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.536050 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.536056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.536205 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.536229 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 21:40:08.536337 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 21:40:08.536477 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"mutex_wait_us":1,"thread_start_us":1087,"threads_started":1}
I20260926 21:40:08.538069 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b14c8415dc4eec9b1637ee84686854" candidate_uuid: "dfc18cce451547b489e8ca8a3da57761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9589bc1c14db43b5a1074273e1f3143f" is_pre_election: true
I20260926 21:40:08.538695 14930 raft_consensus.cc:2454] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc18cce451547b489e8ca8a3da57761 in term 0.
I20260926 21:40:08.538825 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.528444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36053 (local address 127.0.97.2:46211)
0926 21:40:08.530197 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.530214 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:08.530401 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:08.530762 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:08.530769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:08.530802 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:08.531425 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.531444 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.533258 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.533264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.535881 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:08.535899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.537889 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.537895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:08.538113 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.538146 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:40:08.538252 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 21:40:08.538389 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1638,"mutex_wait_us":32,"spinlock_wait_cycles":28928,"thread_start_us":1372,"threads_started":1}
I20260926 21:40:08.538831 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.528410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46211 (local address 127.0.97.3:36053)
0926 21:40:08.528612 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.528619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.530320 (+ 1701us) server_negotiation.cc:407] Connection header received
0926 21:40:08.530545 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.530551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.530634 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.530725 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:08.531690 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.533026 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.536096 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.537628 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.537783 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.537919 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.538294 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.538477 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:40:08.539615 14996 leader_election.cc:302] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9589bc1c14db43b5a1074273e1f3143f, dfc18cce451547b489e8ca8a3da57761; no voters:
I20260926 21:40:08.539826 15193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b14c8415dc4eec9b1637ee84686854" candidate_uuid: "dfc18cce451547b489e8ca8a3da57761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e004c556894462883339583e520e373" is_pre_election: true
I20260926 21:40:08.540031 15260 raft_consensus.cc:2781] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:40:08.540191 15260 raft_consensus.cc:482] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:40:08.540298 15260 raft_consensus.cc:3037] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:08.540427 15193 raft_consensus.cc:2454] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc18cce451547b489e8ca8a3da57761 in term 0.
I20260926 21:40:08.543216 15260 raft_consensus.cc:504] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:08.543919 15260 leader_election.cc:288] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [CANDIDATE]: Term 1 election: Requested vote from peers 7e004c556894462883339583e520e373 (127.0.97.3:36053), 9589bc1c14db43b5a1074273e1f3143f (127.0.97.1:35239)
I20260926 21:40:08.544375 15193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b14c8415dc4eec9b1637ee84686854" candidate_uuid: "dfc18cce451547b489e8ca8a3da57761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e004c556894462883339583e520e373"
I20260926 21:40:08.544591 15193 raft_consensus.cc:3037] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:08.547075 15193 raft_consensus.cc:2454] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc18cce451547b489e8ca8a3da57761 in term 1.
I20260926 21:40:08.547554 14997 leader_election.cc:302] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e004c556894462883339583e520e373, dfc18cce451547b489e8ca8a3da57761; no voters:
I20260926 21:40:08.547550 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b14c8415dc4eec9b1637ee84686854" candidate_uuid: "dfc18cce451547b489e8ca8a3da57761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9589bc1c14db43b5a1074273e1f3143f"
I20260926 21:40:08.547771 14930 raft_consensus.cc:3037] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:08.547938 15260 raft_consensus.cc:2781] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:40:08.548454 15260 raft_consensus.cc:686] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 1 LEADER]: Becoming Leader. State: Replica: dfc18cce451547b489e8ca8a3da57761, State: Running, Role: LEADER
I20260926 21:40:08.549093 15260 consensus_queue.cc:237] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:08.551074 14930 raft_consensus.cc:2454] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc18cce451547b489e8ca8a3da57761 in term 1.
W20260926 21:40:08.557665 14976 tablet.cc:2369] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:40:08.560809 14790 catalog_manager.cc:5495] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfc18cce451547b489e8ca8a3da57761 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "dfc18cce451547b489e8ca8a3da57761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } health_report { overall_health: UNKNOWN } } }
I20260926 21:40:08.616379 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.610662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.0.97.2:44119)
0926 21:40:08.610875 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.610880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.610900 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:40:08.611144 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.611149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.611229 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.611411 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:40:08.611815 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.612548 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.614086 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.614535 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.614695 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.615223 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:40:08.615746 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:40:08.615927 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.615993 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.616079 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:40:08.626003 15193 raft_consensus.cc:1261] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [term 1 FOLLOWER]: Refusing update from remote peer dfc18cce451547b489e8ca8a3da57761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:40:08.627417 14930 raft_consensus.cc:1261] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Refusing update from remote peer dfc18cce451547b489e8ca8a3da57761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:40:08.627180 15265 consensus_queue.cc:1035] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:40:08.628328 15265 consensus_queue.cc:1035] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:40:08.640715 15268 mvcc.cc:205] Tried to move back new op lower bound from 7333719280120664064 to 7333719279822340096. Current Snapshot: MvccSnapshot[applied={T|T < 7333719280120664064}]
I20260926 21:40:08.674360 14790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:40:08.674685 14790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:40:08.679313 14790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Sending DeleteTablet for 3 replicas of tablet e3b14c8415dc4eec9b1637ee84686854
I20260926 21:40:08.680806 15173 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b14c8415dc4eec9b1637ee84686854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37590
I20260926 21:40:08.681355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 14827
I20260926 21:40:08.681476 14910 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b14c8415dc4eec9b1637ee84686854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49666
I20260926 21:40:08.683827 15042 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b14c8415dc4eec9b1637ee84686854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40086
I20260926 21:40:08.684868 14909 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b14c8415dc4eec9b1637ee84686854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49666
I20260926 21:40:08.694998 15279 tablet_replica.cc:330] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: stopping tablet replica
I20260926 21:40:08.695686 15278 tablet_replica.cc:330] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: stopping tablet replica
I20260926 21:40:08.696183 15280 tablet_replica.cc:330] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: stopping tablet replica
I20260926 21:40:08.696615 15280 raft_consensus.cc:2229] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:40:08.697083 15280 raft_consensus.cc:2258] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:08.697746 15279 raft_consensus.cc:2229] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:40:08.698324 15279 raft_consensus.cc:2258] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:08.698930 15278 raft_consensus.cc:2229] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:40:08.699427 15278 raft_consensus.cc:2258] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:08.711838 15278 ts_tablet_manager.cc:1905] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:08.712390 15279 ts_tablet_manager.cc:1905] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:08.714320 15280 ts_tablet_manager.cc:1905] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:08.713004 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.682468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.0.97.1:35239)
0926 21:40:08.682634 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.682641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.682660 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:40:08.682714 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.682719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.682792 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.682905 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:08.685141 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.685998 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.688707 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.689169 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.689352 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.702321 (+ 12969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:08.702337 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:08.702355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:08.702402 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:08.708461 (+ 6059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:08.709630 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:08.709641 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:08.709646 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:08.709839 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:08.710323 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:08.710328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:08.710331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:08.710777 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:08.711076 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.712388 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.712628 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:40:08.715543 14970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:08.721582 15280 ts_tablet_manager.cc:1918] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:08.721588 15279 ts_tablet_manager.cc:1918] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:08.721783 15279 log.cc:1196] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/e3b14c8415dc4eec9b1637ee84686854
I20260926 21:40:08.721776 15280 log.cc:1196] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/e3b14c8415dc4eec9b1637ee84686854
I20260926 21:40:08.722484 15279 ts_tablet_manager.cc:1939] T e3b14c8415dc4eec9b1637ee84686854 P dfc18cce451547b489e8ca8a3da57761: Deleting consensus metadata
I20260926 21:40:08.722708 15280 ts_tablet_manager.cc:1939] T e3b14c8415dc4eec9b1637ee84686854 P 7e004c556894462883339583e520e373: Deleting consensus metadata
I20260926 21:40:08.724846 14776 catalog_manager.cc:4841] TS 7e004c556894462883339583e520e373 (127.0.97.3:36053): tablet e3b14c8415dc4eec9b1637ee84686854 (table test-table [id=2a99b994461549329b740ead9e87fb6d]) successfully deleted
I20260926 21:40:08.727391 14775 catalog_manager.cc:4841] TS dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119): tablet e3b14c8415dc4eec9b1637ee84686854 (table test-table [id=2a99b994461549329b740ead9e87fb6d]) successfully deleted
I20260926 21:40:08.867480 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 14827
I20260926 21:40:08.887322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 14979
W20260926 21:40:08.887485 14775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35239 (error 108)
I20260926 21:40:08.889822 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.889062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56790)
0926 21:40:08.889484 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.889605 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":573,"threads_started":1}
W20260926 21:40:08.890133 14775 catalog_manager.cc:4584] TS 9589bc1c14db43b5a1074273e1f3143f (127.0.97.1:35239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3b14c8415dc4eec9b1637ee84686854: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
I20260926 21:40:08.909539 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.888109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.0.97.2:44119)
0926 21:40:08.888294 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:08.888300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:08.889237 (+ 937us) server_negotiation.cc:407] Connection header received
0926 21:40:08.889319 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:08.889325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:08.889423 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:08.889531 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:08.891298 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.892095 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.894097 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:08.894531 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:08.894630 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:08.903306 (+ 8676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:08.903317 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:08.903332 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:08.903376 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:08.906723 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:08.907531 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:08.907538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:08.907540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:08.907668 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:08.908077 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:08.908080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:08.908082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:08.908410 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:08.908662 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:08.908987 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 21:40:08.909144 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:40:08.910266 15102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:08.937418 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.934217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56806)
0926 21:40:08.937065 (+ 2848us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.937231 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2658}
I20260926 21:40:08.938687 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.938226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56822)
0926 21:40:08.938434 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.938519 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 21:40:08.944051 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.943566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56824)
0926 21:40:08.943789 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.943897 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 21:40:08.945060 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.944667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56826)
0926 21:40:08.944828 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.944908 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 21:40:08.987818 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.987258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56834)
0926 21:40:08.987508 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.987617 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 21:40:08.988978 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:08.988527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56836)
0926 21:40:08.988728 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:08.988813 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 21:40:09.021544 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:09.020915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56838)
0926 21:40:09.021166 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:09.021325 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 21:40:09.022629 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:09.022212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56848)
0926 21:40:09.022390 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:09.022473 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 21:40:09.029516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 14979
I20260926 21:40:09.049463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 15110
I20260926 21:40:09.062448 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:09.050216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.0.97.3:36053)
0926 21:40:09.050707 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:09.050712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:09.050726 (+ 14us) server_negotiation.cc:407] Connection header received
0926 21:40:09.050783 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:09.050787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:09.050847 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:09.050965 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:09.052210 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:09.052909 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:09.054289 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:09.054738 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:09.054875 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:09.057373 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:09.057387 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:09.057403 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:09.057448 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:09.059992 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:09.060667 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:09.060674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:09.060676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:09.060795 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:09.061125 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:09.061128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:09.061129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:09.061526 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:09.061751 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:09.062015 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 21:40:09.062172 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260926 21:40:09.063093 15233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:09.089468 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:09.088995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56852)
0926 21:40:09.089218 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:09.089320 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 21:40:09.090484 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:09.090113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56856)
0926 21:40:09.090279 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:09.090342 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 21:40:09.124779 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:09.124340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56858)
0926 21:40:09.124539 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:09.124642 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 21:40:09.125818 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:09.125448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35239 (local address 127.0.0.1:56874)
0926 21:40:09.125597 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:09.125683 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 21:40:09.152277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 15110
I20260926 21:40:09.169293 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 14757
I20260926 21:40:09.170522 14820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:09.286456 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 14757
I20260926 21:40:09.308064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39563
--webserver_interface=127.0.97.62
--webserver_port=37651
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:09.640532 15288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:09.640892 15288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:09.641052 15288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:09.648406 15288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:40:09.648558 15288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:09.648674 15288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:09.648780 15288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:40:09.648854 15288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:40:09.659677 15288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:09.661115 15288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:09.662515 15288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:09.674352 15293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:09.674393 15294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:09.674466 15296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:09.675732 15288 server_base.cc:1034] running on GCE node
I20260926 21:40:09.676560 15288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:09.678092 15288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:09.679338 15288 hybrid_clock.cc:648] HybridClock initialized: now 1790458809679272 us; error 56 us; skew 500 ppm
I20260926 21:40:09.679715 15288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:09 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:09.683046 15288 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:40:09.684605 15288 webserver.cc:460] Webserver started at http://127.0.97.62:37651/ using document root <none> and password file <none>
I20260926 21:40:09.685395 15288 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:09.685544 15288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:09.691043 15288 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 21:40:09.693821 15303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:09.695124 15288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:40:09.695312 15288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f"
format_stamp: "Formatted at 2026-09-26 21:40:03 on dist-test-slave-lhhq"
server_key: "8bedc6a54d47a77b57f0ffb967c79e9e"
server_key_iv: "8b9c9a52d6e2ea4a0d8fc6e3b2e79fe6"
server_key_version: "encryptionkey@0"
I20260926 21:40:09.696285 15288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:09.730676 15288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:09.734946 15288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:09.735244 15288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:09.754210 15288 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39563
I20260926 21:40:09.754245 15354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39563 every 8 connection(s)
I20260926 21:40:09.755841 15288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:40:09.760077 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 15288
I20260926 21:40:09.761090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35239
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39563
--webserver_port=44721
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:40:09.764465 15355 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:40:09.769757 15355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Bootstrap starting.
I20260926 21:40:09.790490 15355 log.cc:824] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:09.808109 15355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:40:09.808704 15355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f: Bootstrap complete.
I20260926 21:40:09.818279 15355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:09.819267 15355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9a1e52ef61483fa10e6bb8edbd3e4f, State: Initialized, Role: FOLLOWER
I20260926 21:40:09.819968 15355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:09.820207 15355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:40:09.820336 15355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:40:09.820535 15355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 2 FOLLOWER]: Advancing to term 3
I20260926 21:40:09.824581 15355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:09.825124 15355 leader_election.cc:302] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a9a1e52ef61483fa10e6bb8edbd3e4f; no voters:
I20260926 21:40:09.825946 15355 leader_election.cc:288] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 21:40:09.826094 15359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 3 FOLLOWER]: Leader election won for term 3
I20260926 21:40:09.827392 15359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [term 3 LEADER]: Becoming Leader. State: Replica: 8a9a1e52ef61483fa10e6bb8edbd3e4f, State: Running, Role: LEADER
I20260926 21:40:09.828042 15359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } }
I20260926 21:40:09.828383 15355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:40:09.830922 15360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } } }
I20260926 21:40:09.830960 15361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a9a1e52ef61483fa10e6bb8edbd3e4f. Latest consensus state: current_term: 3 leader_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9a1e52ef61483fa10e6bb8edbd3e4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39563 } } }
I20260926 21:40:09.831298 15360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:09.831353 15361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a9a1e52ef61483fa10e6bb8edbd3e4f [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:09.832492 15365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:40:09.849879 15365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:40:09.851913 15365 catalog_manager.cc:1259] Loaded cluster ID: 0e3437b1b9e44d72b0323855d717bdab
I20260926 21:40:09.852124 15365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 21:40:09.854897 15365 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f05494419ad google::LogMessage::Fail() at ??:0
@ 0x7f054944488a google::LogMessage::SendToLog() at ??:0
@ 0x7f05494414de google::LogMessage::Flush() at ??:0
@ 0x7f0549442a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f055ee74512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f055ee71976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f055eef9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f055eef9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f054c08d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f054c09091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20260926 21:40:09.915071 15383 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f054c090799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f054c06beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f054f3456db start_thread at ??:0
@ 0x7f05482db61f clone at ??:0
W20260926 21:40:10.158737 15357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:10.159029 15357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:10.159191 15357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:10.166749 15357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:10.166893 15357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:10.167083 15357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:40:10.177042 15357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44721
--tserver_master_addrs=127.0.97.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:10.178396 15357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:10.179733 15357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:10.192931 15397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:10.193043 15396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:10.193022 15399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:10.193632 15357 server_base.cc:1034] running on GCE node
I20260926 21:40:10.194263 15357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:10.195185 15357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:10.196381 15357 hybrid_clock.cc:648] HybridClock initialized: now 1790458810196328 us; error 62 us; skew 500 ppm
I20260926 21:40:10.196698 15357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:10 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:10.199856 15357 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:40:10.201443 15357 webserver.cc:460] Webserver started at http://127.0.97.1:44721/ using document root <none> and password file <none>
I20260926 21:40:10.202142 15357 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:10.202294 15357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:10.207703 15357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 21:40:10.210705 15406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:10.212060 15357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:40:10.212245 15357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "9589bc1c14db43b5a1074273e1f3143f"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "137970db97df85336ae9209e023910f3"
server_key_iv: "5f36323910becd6e528998ffe413f74f"
server_key_version: "encryptionkey@0"
I20260926 21:40:10.213269 15357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:10.227712 15357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:10.232149 15357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:10.232438 15357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:10.233456 15357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:10.235186 15413 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 21:40:10.240790 15357 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 21:40:10.240926 15357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260926 21:40:10.241057 15357 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 21:40:10.244328 15357 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 21:40:10.244433 15357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260926 21:40:10.244791 15413 tablet_bootstrap.cc:492] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Bootstrap starting.
I20260926 21:40:10.254004 15413 log.cc:824] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:10.273489 15413 tablet_bootstrap.cc:492] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:40:10.274353 15413 tablet_bootstrap.cc:492] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Bootstrap complete.
I20260926 21:40:10.275013 15413 ts_tablet_manager.cc:1397] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.011s
I20260926 21:40:10.282652 15357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35239
I20260926 21:40:10.282677 15520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35239 every 8 connection(s)
I20260926 21:40:10.284369 15357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:40:10.287094 15413 raft_consensus.cc:348] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:10.288211 15413 raft_consensus.cc:729] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9589bc1c14db43b5a1074273e1f3143f, State: Initialized, Role: FOLLOWER
I20260926 21:40:10.288522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 15357
I20260926 21:40:10.289014 15413 consensus_queue.cc:260] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:10.289438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44119
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39563
--webserver_port=39263
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:40:10.290385 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:10.287489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:33483)
0926 21:40:10.288701 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:10.289011 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":450,"threads_started":1}
W20260926 21:40:10.291056 15521 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39563 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39563: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
I20260926 21:40:10.291098 15413 ts_tablet_manager.cc:1428] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 21:40:10.291913 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:10.291575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:57501)
0926 21:40:10.291725 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:10.291813 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 21:40:10.292889 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:10.292432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:49661)
0926 21:40:10.292609 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:10.292710 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 21:40:10.293147 15521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:40:10.627724 15525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:10.628063 15525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:10.628240 15525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:10.635490 15525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:10.635645 15525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:10.635833 15525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:40:10.645733 15525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39263
--tserver_master_addrs=127.0.97.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:10.647024 15525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:10.648355 15525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:10.661540 15530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:10.661502 15531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:10.661571 15533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:10.662348 15525 server_base.cc:1034] running on GCE node
I20260926 21:40:10.662938 15525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:10.663849 15525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:10.665051 15525 hybrid_clock.cc:648] HybridClock initialized: now 1790458810664999 us; error 40 us; skew 500 ppm
I20260926 21:40:10.665387 15525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:10 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:10.668447 15525 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:40:10.670077 15525 webserver.cc:460] Webserver started at http://127.0.97.2:39263/ using document root <none> and password file <none>
I20260926 21:40:10.670754 15525 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:10.670857 15525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:10.676242 15525 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 21:40:10.679179 15540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:10.680495 15525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:40:10.680682 15525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "dfc18cce451547b489e8ca8a3da57761"
format_stamp: "Formatted at 2026-09-26 21:40:04 on dist-test-slave-lhhq"
server_key: "60187e15ec515051951db07ae39b7d97"
server_key_iv: "0e85dedd08e6887d437ba409a53bd154"
server_key_version: "encryptionkey@0"
I20260926 21:40:10.681744 15525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:10.706446 15525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:10.710660 15525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:10.710961 15525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:10.711995 15525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:10.713429 15525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:10.713562 15525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:10.713702 15525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:10.713792 15525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:10.749096 15525 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44119
I20260926 21:40:10.749153 15652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44119 every 8 connection(s)
I20260926 21:40:10.750713 15525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:40:10.753749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 15525
I20260926 21:40:10.754577 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36053
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39563
--webserver_port=40197
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:40:10.756400 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:10.753833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:48173)
0926 21:40:10.754774 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:10.755134 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":328,"threads_started":1}
W20260926 21:40:10.757083 15653 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39563 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39563: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
I20260926 21:40:10.757968 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:10.757629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:51705)
0926 21:40:10.757786 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:10.757860 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 21:40:10.758911 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:10.758577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:34599)
0926 21:40:10.758720 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:10.758797 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 21:40:10.759204 15653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:40:11.085346 15656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:11.085625 15656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:11.085824 15656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:11.093048 15656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:11.093159 15656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:11.093379 15656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:40:11.103255 15656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40197
--tserver_master_addrs=127.0.97.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:11.104583 15656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:11.105985 15656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:11.119668 15664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:11.119781 15662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:11.120395 15656 server_base.cc:1034] running on GCE node
W20260926 21:40:11.120551 15661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:11.121357 15656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:11.122268 15656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:11.123505 15656 hybrid_clock.cc:648] HybridClock initialized: now 1790458811123450 us; error 48 us; skew 500 ppm
I20260926 21:40:11.123822 15656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:11 dist-test-slave-lhhq krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:11.126957 15656 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:40:11.128458 15656 webserver.cc:460] Webserver started at http://127.0.97.3:40197/ using document root <none> and password file <none>
I20260926 21:40:11.129136 15656 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:11.129305 15656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:11.134825 15656 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 21:40:11.137578 15671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:11.138903 15656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:40:11.139071 15656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "7e004c556894462883339583e520e373"
format_stamp: "Formatted at 2026-09-26 21:40:05 on dist-test-slave-lhhq"
server_key: "947c22fd8ba76957c39689a90d199b9f"
server_key_iv: "c3fc73d9ea962704aca6575139860ce8"
server_key_version: "encryptionkey@0"
I20260926 21:40:11.140059 15656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:11.162554 15656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:11.167115 15656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:11.167418 15656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:11.168533 15656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:11.169948 15656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:11.170082 15656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:11.170219 15656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:11.170339 15656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:11.204370 15656 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36053
I20260926 21:40:11.204460 15783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36053 every 8 connection(s)
I20260926 21:40:11.205873 15656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:40:11.209431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 15656
I20260926 21:40:11.211016 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:11.208929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:44545)
0926 21:40:11.209832 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:11.210099 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":397,"threads_started":1}
W20260926 21:40:11.211691 15784 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39563 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39563: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
I20260926 21:40:11.212456 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:11.212170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:59747)
0926 21:40:11.212294 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:11.212359 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 21:40:11.213328 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:11.212996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:36793)
0926 21:40:11.213104 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:11.213171 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 21:40:11.213569 15784 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 21:40:11.294324 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:11.293650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:33335)
0926 21:40:11.294053 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:11.294135 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":197,"threads_started":1}
I20260926 21:40:11.760603 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:11.759765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:38461)
0926 21:40:11.760245 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:11.760337 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260926 21:40:11.999435 15790 raft_consensus.cc:482] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:40:11.999792 15790 raft_consensus.cc:504] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:12.001545 15790 leader_election.cc:288] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e004c556894462883339583e520e373 (127.0.97.3:36053), dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119)
Sep 26 21:40:12 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 26 21:40:12 dist-test-slave-lhhq krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:40:12.024667 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:12.002212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55355 (local address 127.0.97.2:44119)
0926 21:40:12.005059 (+ 2847us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:12.005069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:12.005101 (+ 32us) server_negotiation.cc:407] Connection header received
0926 21:40:12.006616 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:12.006623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:12.007012 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:12.007341 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:12.008930 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:12.010145 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:12.012151 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:12.012777 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:12.012969 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:12.017286 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:12.017294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:12.017306 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:12.017340 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:12.020831 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:12.021632 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:12.021642 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:12.021652 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:12.021886 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:12.022367 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:12.022371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:12.022375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:12.023045 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:12.023230 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:12.023877 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 21:40:12.024241 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2685,"thread_start_us":2538,"threads_started":1}
I20260926 21:40:12.024677 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:12.001922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44119 (local address 127.0.97.1:55355)
0926 21:40:12.004725 (+ 2803us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:12.004745 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:12.004863 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:12.007304 (+ 2441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:12.007307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:12.008055 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:12.008731 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:12.008751 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:12.010331 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:12.010334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:12.012013 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:12.012027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:12.012994 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:12.013000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:12.013320 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:12.014154 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:12.014182 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:12.017001 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:12.020973 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:12.020976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:12.020979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:12.021493 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:12.022036 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:12.022041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:12.022045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:12.022217 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:12.023177 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:12.023182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:12.023389 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 21:40:12.023665 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 21:40:12.024240 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2682,"mutex_wait_us":472,"spinlock_wait_cycles":27392,"thread_start_us":283,"threads_started":1}
I20260926 21:40:12.024760 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:12.002197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60733 (local address 127.0.97.3:36053)
0926 21:40:12.004442 (+ 2245us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:12.004461 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:12.004522 (+ 61us) server_negotiation.cc:407] Connection header received
0926 21:40:12.006182 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:12.006192 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:12.006561 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:12.006822 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:12.008931 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:12.010121 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:12.012153 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:12.012770 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:12.012972 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:12.016981 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:12.016992 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:12.017013 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:12.017059 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:12.020196 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:12.021160 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:12.021169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:12.021177 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:12.021401 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:12.022007 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:12.022014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:12.022017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:12.022740 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:12.022961 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:12.023915 (+ 954us) server_negotiation.cc:299] Negotiation successful
0926 21:40:12.024273 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2018,"thread_start_us":1874,"threads_started":1}
I20260926 21:40:12.024953 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:12.001922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36053 (local address 127.0.97.1:60733)
0926 21:40:12.002817 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:12.002870 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:12.003571 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:12.007011 (+ 3440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:12.007024 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:12.008057 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:12.008731 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:12.008751 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:12.010320 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:12.010325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:12.012014 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:12.012026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:12.012985 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:12.012991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:12.013318 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:12.014063 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:12.014085 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:12.016616 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:12.020368 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:12.020380 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:12.020397 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:12.021014 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:12.021530 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:12.021534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:12.021558 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:12.021820 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:12.023067 (+ 1247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:12.023073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:12.023389 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 21:40:12.023666 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 21:40:12.024417 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":371,"spinlock_wait_cycles":22528,"thread_start_us":2474,"threads_started":1}
W20260926 21:40:12.027999 15409 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e004c556894462883339583e520e373 (127.0.97.3:36053): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
W20260926 21:40:12.028278 15408 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
I20260926 21:40:12.028465 15408 leader_election.cc:302] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9589bc1c14db43b5a1074273e1f3143f; no voters: 7e004c556894462883339583e520e373, dfc18cce451547b489e8ca8a3da57761
I20260926 21:40:12.028971 15790 raft_consensus.cc:2726] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 21:40:12.214849 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:12.214027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:48975)
0926 21:40:12.214475 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:12.214609 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":213,"threads_started":1}
I20260926 21:40:12.295643 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:12.295217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:52325)
0926 21:40:12.295394 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:12.295504 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 21:40:12.762271 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:12.761581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:59129)
0926 21:40:12.762030 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:12.762112 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260926 21:40:13.216461 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:13.215714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:50565)
0926 21:40:13.216202 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:13.216285 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":234,"threads_started":1}
I20260926 21:40:13.297500 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:13.296663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:53029)
0926 21:40:13.297172 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:13.297310 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":294,"threads_started":1}
I20260926 21:40:13.764114 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:13.763283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:34205)
0926 21:40:13.763799 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:13.763890 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":279,"threads_started":1}
I20260926 21:40:13.975144 15800 raft_consensus.cc:482] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:40:13.975361 15800 raft_consensus.cc:504] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:13.976034 15800 leader_election.cc:288] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e004c556894462883339583e520e373 (127.0.97.3:36053), dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119)
W20260926 21:40:13.977125 15409 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e004c556894462883339583e520e373 (127.0.97.3:36053): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
W20260926 21:40:13.977367 15408 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
I20260926 21:40:13.977500 15408 leader_election.cc:302] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9589bc1c14db43b5a1074273e1f3143f; no voters: 7e004c556894462883339583e520e373, dfc18cce451547b489e8ca8a3da57761
I20260926 21:40:13.977752 15800 raft_consensus.cc:2726] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 21:40:14.218072 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:14.217369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:52713)
0926 21:40:14.217815 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:14.217889 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260926 21:40:14.299152 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:14.298383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:38023)
0926 21:40:14.298864 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:14.298953 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":229,"threads_started":1}
I20260926 21:40:14.765831 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:14.765081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:36795)
0926 21:40:14.765571 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:14.765646 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":272,"threads_started":1}
I20260926 21:40:15.219815 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:15.218994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:49567)
0926 21:40:15.219417 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:15.219533 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":210,"threads_started":1}
I20260926 21:40:15.300963 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:15.300223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:38965)
0926 21:40:15.300693 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:15.300771 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":221,"threads_started":1}
I20260926 21:40:15.767707 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:15.766932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:40025)
0926 21:40:15.767416 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:15.767503 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":260,"threads_started":1}
I20260926 21:40:16.221467 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:16.220741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:37455)
0926 21:40:16.221168 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:16.221303 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":236,"threads_started":1}
I20260926 21:40:16.302675 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:16.301952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:59403)
0926 21:40:16.302409 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:16.302514 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":227,"threads_started":1}
I20260926 21:40:16.769553 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:16.768675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:47369)
0926 21:40:16.769247 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:16.769336 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":297,"threads_started":1}
I20260926 21:40:17.203725 15810 raft_consensus.cc:482] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:40:17.203893 15810 raft_consensus.cc:504] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:17.204594 15810 leader_election.cc:288] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e004c556894462883339583e520e373 (127.0.97.3:36053), dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119)
W20260926 21:40:17.205559 15408 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
W20260926 21:40:17.205781 15409 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e004c556894462883339583e520e373 (127.0.97.3:36053): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
I20260926 21:40:17.205915 15409 leader_election.cc:302] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9589bc1c14db43b5a1074273e1f3143f; no voters: 7e004c556894462883339583e520e373, dfc18cce451547b489e8ca8a3da57761
I20260926 21:40:17.206169 15810 raft_consensus.cc:2726] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 21:40:17.223199 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:17.222444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:57349)
0926 21:40:17.222918 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:17.222991 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260926 21:40:17.304368 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:17.303563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:50833)
0926 21:40:17.304046 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:17.304159 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":235,"threads_started":1}
I20260926 21:40:17.771452 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:17.770638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:50433)
0926 21:40:17.771145 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:17.771242 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":229,"threads_started":1}
I20260926 21:40:18.224970 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:18.224199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:53415)
0926 21:40:18.224708 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:18.224789 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":212,"threads_started":1}
I20260926 21:40:18.306165 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:18.305448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:45999)
0926 21:40:18.305908 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:18.306005 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":250,"threads_started":1}
I20260926 21:40:18.773742 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:18.772469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:37197)
0926 21:40:18.773334 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:18.773461 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":303,"threads_started":1}
I20260926 21:40:18.977161 15817 raft_consensus.cc:482] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:40:18.977425 15817 raft_consensus.cc:504] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:18.978089 15817 leader_election.cc:288] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e004c556894462883339583e520e373 (127.0.97.3:36053), dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119)
W20260926 21:40:18.979125 15409 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e004c556894462883339583e520e373 (127.0.97.3:36053): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
W20260926 21:40:18.979357 15408 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
I20260926 21:40:18.979542 15408 leader_election.cc:302] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9589bc1c14db43b5a1074273e1f3143f; no voters: 7e004c556894462883339583e520e373, dfc18cce451547b489e8ca8a3da57761
I20260926 21:40:18.979830 15817 raft_consensus.cc:2726] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 21:40:19.226779 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:19.226037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:50159)
0926 21:40:19.226505 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:19.226597 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
I20260926 21:40:19.308158 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:19.307230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:40921)
0926 21:40:19.307835 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:19.307916 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":283,"threads_started":1}
I20260926 21:40:19.775667 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:19.774829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:44701)
0926 21:40:19.775378 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:19.775490 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":245,"threads_started":1}
I20260926 21:40:20.228716 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:20.227794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:60345)
0926 21:40:20.228404 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:20.228512 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":283,"threads_started":1}
I20260926 21:40:20.309876 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:20.309122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:52793)
0926 21:40:20.309617 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:20.309704 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":289,"threads_started":1}
I20260926 21:40:20.777555 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:20.776635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:40605)
0926 21:40:20.777161 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:20.777352 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":295,"threads_started":1}
I20260926 21:40:21.230562 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:21.229721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:58261)
0926 21:40:21.230270 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:21.230354 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":252,"threads_started":1}
I20260926 21:40:21.288019 15825 raft_consensus.cc:482] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:40:21.288592 15825 raft_consensus.cc:504] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e004c556894462883339583e520e373" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36053 } } peers { permanent_uuid: "dfc18cce451547b489e8ca8a3da57761" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44119 } } peers { permanent_uuid: "9589bc1c14db43b5a1074273e1f3143f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35239 } }
I20260926 21:40:21.289301 15825 leader_election.cc:288] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e004c556894462883339583e520e373 (127.0.97.3:36053), dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119)
W20260926 21:40:21.290252 15409 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e004c556894462883339583e520e373 (127.0.97.3:36053): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
W20260926 21:40:21.290450 15408 leader_election.cc:341] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfc18cce451547b489e8ca8a3da57761 (127.0.97.2:44119): Not found: Tablet not found: e3b14c8415dc4eec9b1637ee84686854
I20260926 21:40:21.290632 15408 leader_election.cc:302] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9589bc1c14db43b5a1074273e1f3143f; no voters: 7e004c556894462883339583e520e373, dfc18cce451547b489e8ca8a3da57761
I20260926 21:40:21.290911 15825 raft_consensus.cc:2726] T e3b14c8415dc4eec9b1637ee84686854 P 9589bc1c14db43b5a1074273e1f3143f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 21:40:21.311575 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:21.310737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:43775)
0926 21:40:21.311264 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:21.311341 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":263,"threads_started":1}
I20260926 21:40:21.779414 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:21.778634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:51135)
0926 21:40:21.779154 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:21.779242 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":243,"threads_started":1}
I20260926 21:40:22.232331 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:22.231536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:53389)
0926 21:40:22.232087 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:22.232170 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":300,"threads_started":1}
I20260926 21:40:22.313385 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:22.312632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:44643)
0926 21:40:22.313100 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:22.313220 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":229,"threads_started":1}
I20260926 21:40:22.781320 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:22.780475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:54695)
0926 21:40:22.780968 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:22.781051 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":265,"threads_started":1}
I20260926 21:40:23.234283 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:23.233492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:41665)
0926 21:40:23.233981 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:23.234101 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
I20260926 21:40:23.315379 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:23.314638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:54323)
0926 21:40:23.315114 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:23.315208 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":235,"threads_started":1}
I20260926 21:40:23.783231 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:23.782384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:51861)
0926 21:40:23.782943 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:23.783035 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":301,"threads_started":1}
I20260926 21:40:24.236089 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:24.235365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:46967)
0926 21:40:24.235834 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:24.235923 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":235,"threads_started":1}
I20260926 21:40:24.317379 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:24.316551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:58757)
0926 21:40:24.317065 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:24.317169 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":254,"threads_started":1}
I20260926 21:40:24.785418 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:24.784399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:33971)
0926 21:40:24.785010 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:24.785112 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":355,"threads_started":1}
I20260926 21:40:25.237882 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:25.237084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:55587)
0926 21:40:25.237567 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:25.237649 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":255,"threads_started":1}
I20260926 21:40:25.319406 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:25.318648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.1:48809)
0926 21:40:25.319148 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:25.319236 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":235,"threads_started":1}
I20260926 21:40:25.787388 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:25.786618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.2:36337)
0926 21:40:25.787127 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:25.787227 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":275,"threads_started":1}
I20260926 21:40:26.214905 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 15357
I20260926 21:40:26.239763 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:26.239016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39563 (local address 127.0.97.3:48439)
0926 21:40:26.239505 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:26.239590 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":268,"threads_started":1}
I20260926 21:40:26.243630 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:26.216184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.0.97.1:35239)
0926 21:40:26.216749 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:26.216795 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:26.216829 (+ 34us) server_negotiation.cc:407] Connection header received
0926 21:40:26.216928 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:26.216937 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:26.217278 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:26.217472 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:26.219547 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:26.220504 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:26.221832 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:26.222418 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:26.222519 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:26.237624 (+ 15105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:26.237640 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:26.237657 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:26.237696 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:26.240942 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:26.241715 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:26.241735 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:26.241737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:26.241883 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:26.242231 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:26.242235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:26.242237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:26.242622 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:26.242862 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:26.243082 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 21:40:26.243298 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":257,"threads_started":1}
I20260926 21:40:26.245229 15516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:26.337687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 15357
I20260926 21:40:26.356727 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 15525
I20260926 21:40:26.369976 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:26.357630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.0.97.2:44119)
0926 21:40:26.358190 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:26.358194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:26.358211 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:40:26.358265 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:26.358268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:26.358328 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:26.358422 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:26.359689 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:26.360492 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:26.361771 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:26.362223 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:26.362332 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:26.364829 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:26.364834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:26.364837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:26.364872 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:26.367681 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:26.368304 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:26.368309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:26.368311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:26.368440 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:26.368744 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:26.368747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:26.368749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:26.369013 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:26.369293 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:26.369488 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 21:40:26.369681 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":310,"threads_started":1}
I20260926 21:40:26.370713 15648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:26.457805 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 15525
I20260926 21:40:26.475975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 15656
I20260926 21:40:26.490202 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:26.476858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.0.97.3:36053)
0926 21:40:26.477442 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:26.477447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:26.477462 (+ 15us) server_negotiation.cc:407] Connection header received
0926 21:40:26.477514 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:26.477517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:26.477574 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:26.477713 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:26.479072 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:26.479880 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:26.481180 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:26.481675 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:26.481831 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:26.484479 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:26.484484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:26.484487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:26.484520 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:26.487250 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:26.487985 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:26.487997 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:26.488000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:26.488143 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:26.488535 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:26.488542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:26.488545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:26.488821 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:26.489126 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:26.489675 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 21:40:26.489907 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":265,"threads_started":1}
I20260926 21:40:26.490943 15779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:26.575349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 15656
2026-09-26T21:40:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24728 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:40:26 dist-test-slave-lhhq krb5kdc[15850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:40:26 dist-test-slave-lhhq krb5kdc[15850](info): set up 2 sockets
Sep 26 21:40:26 dist-test-slave-lhhq krb5kdc[15850](info): commencing operation
krb5kdc: starting...
W20260926 21:40:27.677834 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:40:27 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:40:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:40:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:40:27.925416 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:28.260597 15863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:28.260905 15863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:28.260995 15863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:28.268312 15863 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 21:40:28.268426 15863 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 21:40:28.268464 15863 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 21:40:28.268513 15863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:40:28.268566 15863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:28.268628 15863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:28.268674 15863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:40:28.268720 15863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:40:28.278848 15863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:28.280135 15863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:28.281422 15863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:28.293927 15869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:28.293990 15868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:28.294015 15871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:28.294677 15863 server_base.cc:1034] running on GCE node
I20260926 21:40:28.295509 15863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:28.296954 15863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:28.298213 15863 hybrid_clock.cc:648] HybridClock initialized: now 1790458828298150 us; error 39 us; skew 500 ppm
I20260926 21:40:28.298575 15863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:28 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:28.302098 15863 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:40:28.303933 15863 webserver.cc:460] Webserver started at http://127.0.97.62:45271/ using document root <none> and password file <none>
I20260926 21:40:28.304774 15863 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:28.304939 15863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:28.305406 15863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:40:28.309128 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "0f105a9cf0e649a1a9dff8026d044565"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "d868388eb411901894d5df7384f522c2"
server_key_iv: "6936e7d0193938814bdb48547dd83fe8"
server_key_version: "encryptionkey@0"
I20260926 21:40:28.310019 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f105a9cf0e649a1a9dff8026d044565"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "d868388eb411901894d5df7384f522c2"
server_key_iv: "6936e7d0193938814bdb48547dd83fe8"
server_key_version: "encryptionkey@0"
I20260926 21:40:28.315781 15863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:40:28.319862 15878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:28.321374 15863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:40:28.321573 15863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "0f105a9cf0e649a1a9dff8026d044565"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "d868388eb411901894d5df7384f522c2"
server_key_iv: "6936e7d0193938814bdb48547dd83fe8"
server_key_version: "encryptionkey@0"
I20260926 21:40:28.321831 15863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:28.344978 15863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:28.349968 15863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:28.350263 15863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:28.369680 15863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42309
I20260926 21:40:28.369719 15929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42309 every 8 connection(s)
I20260926 21:40:28.371098 15863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:40:28.375476 15930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:28.379676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 15863
I20260926 21:40:28.379951 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:40:28.380954 388 external_mini_cluster.cc:1428] Setting key f24212a49e3bba32befff559aedf08e8
I20260926 21:40:28.389837 15930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Bootstrap starting.
I20260926 21:40:28.392874 15930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:28.393975 15930 log.cc:824] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Log is configured to *not* fsync() on all Append() calls
Sep 26 21:40:28 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:28.397583 15930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: No bootstrap required, opened a new log
I20260926 21:40:28.406374 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:28.383512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.0.97.62:42309)
0926 21:40:28.384417 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:28.384439 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:28.384483 (+ 44us) server_negotiation.cc:407] Connection header received
0926 21:40:28.386315 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:28.386339 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:28.386811 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:28.387252 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:28.388782 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:28.390248 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:28.391647 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:28.392316 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:28.392520 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:28.397683 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:28.397700 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:28.397730 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:28.397772 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:28.401733 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:28.402697 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:28.402704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:28.402711 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:28.402897 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:28.403400 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:28.403408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:28.403410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:28.403992 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:28.404235 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:28.404728 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 21:40:28.405210 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":459,"threads_started":1}
I20260926 21:40:28.406785 15930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:40:28.407172 15930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:28.407289 15930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f105a9cf0e649a1a9dff8026d044565, State: Initialized, Role: FOLLOWER
I20260926 21:40:28.407850 15930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:40:28.408172 15930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:40:28.408351 15930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:40:28.408588 15930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:28.411015 15930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:40:28.411504 15930 leader_election.cc:302] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f105a9cf0e649a1a9dff8026d044565; no voters:
I20260926 21:40:28.412175 15930 leader_election.cc:288] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:40:28.412318 15935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:40:28.413765 15935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 1 LEADER]: Becoming Leader. State: Replica: 0f105a9cf0e649a1a9dff8026d044565, State: Running, Role: LEADER
I20260926 21:40:28.414407 15930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:40:28.414402 15935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:40:28.417470 15937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f105a9cf0e649a1a9dff8026d044565. Latest consensus state: current_term: 1 leader_uuid: "0f105a9cf0e649a1a9dff8026d044565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260926 21:40:28.417873 15937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:28.417753 15936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f105a9cf0e649a1a9dff8026d044565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260926 21:40:28.418045 15936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:28.420245 15944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:40:28.424571 15944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:40:28.433822 15944 catalog_manager.cc:1347] Generated new cluster ID: 997f8bc0e4f34d32879fcbb7afeba986
I20260926 21:40:28.433947 15944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:40:28.445523 15944 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:40:28.447496 15944 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:40:28.462977 15944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Generated new TSK 0
I20260926 21:40:28.464214 15944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:40:28.532457 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:28.880666 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:28.880993 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:28.881176 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:28.888798 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:28.888969 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:28.889174 15956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:40:28.899317 15956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:28.900650 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:28.902096 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:28.915266 15964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:28.915298 15961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:28.915453 15962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:28.916405 15956 server_base.cc:1034] running on GCE node
I20260926 21:40:28.917137 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:28.918184 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:28.919426 15956 hybrid_clock.cc:648] HybridClock initialized: now 1790458828919390 us; error 35 us; skew 500 ppm
I20260926 21:40:28.919765 15956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:28 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:28.923095 15956 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:40:28.924708 15956 webserver.cc:460] Webserver started at http://127.0.97.1:40035/ using document root <none> and password file <none>
I20260926 21:40:28.925590 15956 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:28.925763 15956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:28.926131 15956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:40:28.929349 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "022373814611430b8100c190aa93364a"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "7b20244ab197311986371ca44cab5980"
server_key_iv: "84d53bfe57911225786e4e7994e597e7"
server_key_version: "encryptionkey@0"
I20260926 21:40:28.930167 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "022373814611430b8100c190aa93364a"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "7b20244ab197311986371ca44cab5980"
server_key_iv: "84d53bfe57911225786e4e7994e597e7"
server_key_version: "encryptionkey@0"
I20260926 21:40:28.935704 15956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:40:28.939657 15971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:28.941080 15956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:40:28.941296 15956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "022373814611430b8100c190aa93364a"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "7b20244ab197311986371ca44cab5980"
server_key_iv: "84d53bfe57911225786e4e7994e597e7"
server_key_version: "encryptionkey@0"
I20260926 21:40:28.941512 15956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:28.955009 15956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:28.960289 15956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:28.960623 15956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:28.961736 15956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:28.963436 15956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:28.963508 15956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:28.963606 15956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:28.963655 15956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:28.995194 15956 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39055
I20260926 21:40:28.995306 16083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39055 every 8 connection(s)
I20260926 21:40:28.996407 15956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:40:28.998615 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 15956
I20260926 21:40:28.998854 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:40:28.999794 388 external_mini_cluster.cc:1428] Setting key 510a0e609bbd1b33ac1d368e668173aa
Sep 26 21:40:29 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:29.019382 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:28.999186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45571 (local address 127.0.97.62:42309)
0926 21:40:28.999651 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:28.999656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:29.001098 (+ 1442us) server_negotiation.cc:407] Connection header received
0926 21:40:29.003173 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:29.003178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:29.003259 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:29.003385 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:29.004888 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:29.005661 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:29.007238 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:29.007647 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:29.007778 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:29.011882 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:29.011892 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:29.011899 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:29.011946 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:29.015878 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:29.016713 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:29.016721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:29.016727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:29.016878 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:29.017330 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:29.017339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:29.017345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:29.017764 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:29.017915 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:29.018616 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 21:40:29.018882 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":235,"threads_started":1}
I20260926 21:40:29.020090 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:28.999999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:45571)
0926 21:40:29.000932 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:29.000970 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:29.002894 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:29.003629 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:29.003637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:29.004151 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:29.004718 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:29.004735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:29.005850 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:29.005857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:29.007094 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:29.007105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:29.007868 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:29.007877 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:29.008132 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:29.008842 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:29.008863 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:29.011580 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:29.016104 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:29.016115 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:29.016134 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:29.016584 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:29.016987 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:29.016990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:29.016992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:29.017179 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:29.017885 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:29.017891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:29.018227 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 21:40:29.018479 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 21:40:29.018953 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":386,"threads_started":1}
I20260926 21:40:29.021720 16084 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260926 21:40:29.022049 16084 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:29.022775 16084 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260926 21:40:29.024855 15895 ts_manager.cc:194] Registered new tserver with Master: 022373814611430b8100c190aa93364a (127.0.97.1:39055)
I20260926 21:40:29.026474 15895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45571
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:40:29.056915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:29.402541 16090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:29.402824 16090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:29.402920 16090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:29.410286 16090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:29.410398 16090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:29.410511 16090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:40:29.420779 16090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:29.422185 16090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:29.423454 16090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:29.436767 16095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:29.436858 16098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:29.436801 16096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:29.437707 16090 server_base.cc:1034] running on GCE node
I20260926 21:40:29.438297 16090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:29.439221 16090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:29.440421 16090 hybrid_clock.cc:648] HybridClock initialized: now 1790458829440370 us; error 36 us; skew 500 ppm
I20260926 21:40:29.440744 16090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:29 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:29.443833 16090 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:40:29.445393 16090 webserver.cc:460] Webserver started at http://127.0.97.2:37731/ using document root <none> and password file <none>
I20260926 21:40:29.446049 16090 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:29.446185 16090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:29.446522 16090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:40:29.449397 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "4344e04c0752453f98d824535198e76b"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "0db0e9c94b4bca5e5cdfd96c6cdb996e"
server_key_iv: "cb0aace4191034b031856059e53e8632"
server_key_version: "encryptionkey@0"
I20260926 21:40:29.450146 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4344e04c0752453f98d824535198e76b"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "0db0e9c94b4bca5e5cdfd96c6cdb996e"
server_key_iv: "cb0aace4191034b031856059e53e8632"
server_key_version: "encryptionkey@0"
I20260926 21:40:29.455166 16090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:40:29.458716 16105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:29.460171 16090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:40:29.460353 16090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "4344e04c0752453f98d824535198e76b"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "0db0e9c94b4bca5e5cdfd96c6cdb996e"
server_key_iv: "cb0aace4191034b031856059e53e8632"
server_key_version: "encryptionkey@0"
I20260926 21:40:29.460537 16090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:29.482465 16090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:29.487123 16090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:29.487529 16090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:29.488624 16090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:29.490125 16090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:29.490228 16090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:29.490365 16090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:29.490432 16090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:29.524482 16090 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39097
I20260926 21:40:29.524573 16217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39097 every 8 connection(s)
I20260926 21:40:29.525991 16090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:40:29.530933 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 16090
I20260926 21:40:29.531144 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:40:29.531591 388 external_mini_cluster.cc:1428] Setting key 279ac3e36161e07476f5f34646f1b344
Sep 26 21:40:29 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:29.548041 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:29.528712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51833 (local address 127.0.97.62:42309)
0926 21:40:29.529164 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:29.529170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:29.530305 (+ 1135us) server_negotiation.cc:407] Connection header received
0926 21:40:29.532352 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:29.532356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:29.532423 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:29.532513 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:29.534221 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:29.535485 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:29.537230 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:29.538009 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:29.538161 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:29.542042 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:29.542047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:29.542051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:29.542084 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:29.544775 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:29.545674 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:29.545678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:29.545681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:29.545802 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:29.546259 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:29.546262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:29.546264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:29.546562 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:29.546725 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:29.547420 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 21:40:29.547659 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260926 21:40:29.549059 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:29.529370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:51833)
0926 21:40:29.530137 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:29.530175 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:29.532061 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:29.532782 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:29.532791 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:29.533467 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:29.534033 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:29.534049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:29.535691 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:29.535696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:29.537040 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:29.537055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:29.538243 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:29.538248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:29.538454 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:29.539210 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:29.539243 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:29.541802 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:29.544910 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:29.544919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:29.544938 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:29.545536 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:29.545912 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:29.545918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:29.545921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:29.546110 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:29.546699 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:29.546707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:29.547063 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 21:40:29.547334 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 21:40:29.547893 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":301,"threads_started":1}
I20260926 21:40:29.550894 16218 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260926 21:40:29.551213 16218 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:29.552023 16218 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260926 21:40:29.553763 15895 ts_manager.cc:194] Registered new tserver with Master: 4344e04c0752453f98d824535198e76b (127.0.97.2:39097)
I20260926 21:40:29.555099 15895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51833
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:40:29.587019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:29.924434 16224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:29.924763 16224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:29.924922 16224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:29.932287 16224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:29.932448 16224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:29.932636 16224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:40:29.942557 16224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:29.943892 16224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:29.945325 16224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:29.958312 16229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:29.958477 16232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:29.958986 16230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:29.959359 16224 server_base.cc:1034] running on GCE node
I20260926 21:40:29.960054 16224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:29.961223 16224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:29.962430 16224 hybrid_clock.cc:648] HybridClock initialized: now 1790458829962366 us; error 54 us; skew 500 ppm
I20260926 21:40:29.962805 16224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:29 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:29.965951 16224 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:40:29.967319 16224 webserver.cc:460] Webserver started at http://127.0.97.3:43243/ using document root <none> and password file <none>
I20260926 21:40:29.967962 16224 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:29.968066 16224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:29.968327 16224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:40:29.971266 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "b4266a1190644edbb96dabe8d7869715"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "5b2c49cd5d03a713729e4be6437e7e6e"
server_key_iv: "0157526aa0e5d501b152ac78af96ae1c"
server_key_version: "encryptionkey@0"
I20260926 21:40:29.971974 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b4266a1190644edbb96dabe8d7869715"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "5b2c49cd5d03a713729e4be6437e7e6e"
server_key_iv: "0157526aa0e5d501b152ac78af96ae1c"
server_key_version: "encryptionkey@0"
I20260926 21:40:29.977020 16224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:40:29.980399 16239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:29.981819 16224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:40:29.981953 16224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "b4266a1190644edbb96dabe8d7869715"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "5b2c49cd5d03a713729e4be6437e7e6e"
server_key_iv: "0157526aa0e5d501b152ac78af96ae1c"
server_key_version: "encryptionkey@0"
I20260926 21:40:29.982124 16224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:30.005012 16224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:30.009011 16224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:30.009335 16224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:30.010252 16224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:30.011480 16224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:30.011555 16224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:30.011638 16224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:30.011682 16224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:30.029399 16084 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
I20260926 21:40:30.049329 16224 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39485
I20260926 21:40:30.049413 16351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39485 every 8 connection(s)
I20260926 21:40:30.050721 16224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:40:30.051781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 16224
I20260926 21:40:30.052003 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:40:30.052469 388 external_mini_cluster.cc:1428] Setting key 710663e777298d3958b461cc69545444
Sep 26 21:40:30 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:30.071805 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.053596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57375 (local address 127.0.97.62:42309)
0926 21:40:30.054180 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.054187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.055388 (+ 1201us) server_negotiation.cc:407] Connection header received
0926 21:40:30.057318 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.057322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.057395 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.057484 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:30.058898 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.059644 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.061072 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.061519 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.061614 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.064990 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:30.064996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:30.064999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:30.065033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:30.068727 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.069603 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.069607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.069610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.069737 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.070157 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.070160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.070162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.070445 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:30.070563 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.071286 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.071497 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":327,"threads_started":1}
I20260926 21:40:30.072806 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.054362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:57375)
0926 21:40:30.055229 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:30.055293 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:30.057036 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:30.057685 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:30.057693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:30.058208 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:30.058744 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.058764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.059762 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.059766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.060957 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.060969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.061700 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.061705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.061906 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.062496 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:30.062517 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:30.064774 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:30.068882 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:30.068890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:30.068904 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:30.069475 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:30.069837 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:30.069840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:30.069843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:30.070023 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:30.070558 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:30.070564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:30.070906 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 21:40:30.071167 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 21:40:30.071573 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":305,"threads_started":1}
I20260926 21:40:30.074354 16352 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260926 21:40:30.074679 16352 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:30.075404 16352 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260926 21:40:30.076874 15895 ts_manager.cc:194] Registered new tserver with Master: b4266a1190644edbb96dabe8d7869715 (127.0.97.3:39485)
I20260926 21:40:30.077746 15895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57375
I20260926 21:40:30.081030 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:40:30.099351 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.086002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.0.97.62:42309)
0926 21:40:30.086270 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.086275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.086591 (+ 316us) server_negotiation.cc:407] Connection header received
0926 21:40:30.086729 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.086739 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.086820 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.086912 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:30.088366 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.089166 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.090741 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.091201 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.091313 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.093991 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:30.093996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:30.093999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:30.094036 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:30.096767 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.097544 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.097548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.097551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.097684 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.098025 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.098028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.098030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.098381 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:30.098561 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.098892 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.099106 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 21:40:30.103766 15895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:40:30.110682 15895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:40:30.136732 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.124721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.0.97.2:39097)
0926 21:40:30.126280 (+ 1559us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.126291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.126992 (+ 701us) server_negotiation.cc:407] Connection header received
0926 21:40:30.127094 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.127102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.127368 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.127568 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:30.128581 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.130262 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.133990 (+ 3728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.135535 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.135687 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.135898 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.136184 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.136370 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1322,"thread_start_us":1403,"threads_started":1}
I20260926 21:40:30.136965 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.123925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.0.97.1:39055)
0926 21:40:30.125215 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.125227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.126145 (+ 918us) server_negotiation.cc:407] Connection header received
0926 21:40:30.126456 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.126466 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.127819 (+ 1353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.128003 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:30.128914 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.131597 (+ 2683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.134331 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.135830 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.135950 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.136184 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.136435 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.136630 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1079,"thread_start_us":925,"threads_started":1}
I20260926 21:40:30.137027 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.123706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39055 (local address 127.0.0.1:38818)
0926 21:40:30.125626 (+ 1920us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:30.125669 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:30.125863 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:30.128241 (+ 2378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:30.128245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:30.128271 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:30.128718 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.128730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.131770 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.131775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.134180 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.134193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.136054 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.136063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.136245 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.136277 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 21:40:30.136367 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 21:40:30.136580 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1658,"mutex_wait_us":2,"thread_start_us":1513,"threads_started":1}
I20260926 21:40:30.137570 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.124079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39097 (local address 127.0.0.1:60672)
0926 21:40:30.126727 (+ 2648us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:30.126744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:30.126861 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:30.127870 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:30.127878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:30.127913 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:30.128370 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.128387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.130436 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.130442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.133822 (+ 3380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.133836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.135718 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.135724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.135898 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.135965 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 21:40:30.136950 (+ 985us) client_negotiation.cc:240] Negotiation successful
0926 21:40:30.137111 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2483,"mutex_wait_us":797,"thread_start_us":2343,"threads_started":1}
I20260926 21:40:30.141275 16019 tablet_service.cc:1467] Processing CreateTablet for tablet 793ca6f46e274f42b2e0dbc69bde3934 (DEFAULT_TABLE table=test-table [id=8330d5252a2f4e38a9b6b1f246a86a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:30.141279 16153 tablet_service.cc:1467] Processing CreateTablet for tablet 793ca6f46e274f42b2e0dbc69bde3934 (DEFAULT_TABLE table=test-table [id=8330d5252a2f4e38a9b6b1f246a86a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:30.142093 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.125982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39485 (local address 127.0.0.1:56670)
0926 21:40:30.132857 (+ 6875us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:30.132881 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:30.133003 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:30.133916 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:30.133921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:30.133947 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:30.134398 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.134413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.136296 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.136300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.139320 (+ 3020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.139335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.141142 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.141149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.141357 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.141392 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:40:30.141493 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 21:40:30.141620 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6738,"thread_start_us":6598,"threads_started":1}
I20260926 21:40:30.142133 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.126351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.0.97.3:39485)
0926 21:40:30.133445 (+ 7094us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.133451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.133477 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:40:30.133552 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.133557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.133734 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.133992 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:30.134615 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.136146 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.139531 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.140920 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.141069 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.141346 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.141555 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.141724 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6919,"thread_start_us":6725,"threads_started":1}
I20260926 21:40:30.142824 16153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793ca6f46e274f42b2e0dbc69bde3934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:30.142833 16019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793ca6f46e274f42b2e0dbc69bde3934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:30.146351 16287 tablet_service.cc:1467] Processing CreateTablet for tablet 793ca6f46e274f42b2e0dbc69bde3934 (DEFAULT_TABLE table=test-table [id=8330d5252a2f4e38a9b6b1f246a86a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:30.147799 16287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793ca6f46e274f42b2e0dbc69bde3934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:30.154953 16371 tablet_bootstrap.cc:492] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: Bootstrap starting.
I20260926 21:40:30.157943 16372 tablet_bootstrap.cc:492] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: Bootstrap starting.
I20260926 21:40:30.158552 16371 tablet_bootstrap.cc:654] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:30.159924 16371 log.cc:824] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:30.161451 16372 tablet_bootstrap.cc:654] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:30.162688 16372 log.cc:824] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:30.161700 16374 tablet_bootstrap.cc:492] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: Bootstrap starting.
I20260926 21:40:30.163103 16371 tablet_bootstrap.cc:492] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: No bootstrap required, opened a new log
I20260926 21:40:30.163386 16371 ts_tablet_manager.cc:1397] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 21:40:30.165936 16372 tablet_bootstrap.cc:492] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: No bootstrap required, opened a new log
I20260926 21:40:30.165998 16374 tablet_bootstrap.cc:654] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:30.166206 16372 ts_tablet_manager.cc:1397] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260926 21:40:30.167237 16374 log.cc:824] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:30.170276 16374 tablet_bootstrap.cc:492] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: No bootstrap required, opened a new log
I20260926 21:40:30.170531 16374 ts_tablet_manager.cc:1397] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 21:40:30.171567 16371 raft_consensus.cc:348] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } }
I20260926 21:40:30.171931 16371 raft_consensus.cc:374] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:30.172065 16371 raft_consensus.cc:729] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022373814611430b8100c190aa93364a, State: Initialized, Role: FOLLOWER
I20260926 21:40:30.172755 16371 consensus_queue.cc:260] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } }
I20260926 21:40:30.174695 16371 ts_tablet_manager.cc:1428] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260926 21:40:30.176188 16372 raft_consensus.cc:348] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } }
I20260926 21:40:30.176561 16372 raft_consensus.cc:374] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:30.176664 16372 raft_consensus.cc:729] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4344e04c0752453f98d824535198e76b, State: Initialized, Role: FOLLOWER
I20260926 21:40:30.177304 16372 consensus_queue.cc:260] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } }
I20260926 21:40:30.178390 16218 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
I20260926 21:40:30.179076 16372 ts_tablet_manager.cc:1428] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 21:40:30.182322 16374 raft_consensus.cc:348] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } }
I20260926 21:40:30.182819 16374 raft_consensus.cc:374] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:30.182930 16374 raft_consensus.cc:729] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4266a1190644edbb96dabe8d7869715, State: Initialized, Role: FOLLOWER
I20260926 21:40:30.183533 16374 consensus_queue.cc:260] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } }
I20260926 21:40:30.184517 16352 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
I20260926 21:40:30.184988 16374 ts_tablet_manager.cc:1428] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 21:40:30.229867 16379 raft_consensus.cc:482] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:40:30.230201 16379 raft_consensus.cc:504] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } }
I20260926 21:40:30.231922 16379 leader_election.cc:288] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 022373814611430b8100c190aa93364a (127.0.97.1:39055), 4344e04c0752453f98d824535198e76b (127.0.97.2:39097)
I20260926 21:40:30.237434 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.232523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33211 (local address 127.0.97.1:39055)
0926 21:40:30.232723 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.232730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.232756 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:40:30.232832 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.232844 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.232902 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.233009 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:30.233461 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.234179 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.235747 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.236533 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.236690 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.236792 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.237036 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.237166 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 21:40:30.237450 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.232237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39055 (local address 127.0.97.3:33211)
0926 21:40:30.232545 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:30.232568 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:30.232699 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:30.233034 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:30.233038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:30.233075 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:30.233321 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.233331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.234312 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.234318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.235587 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.235599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.236698 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.236703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.236884 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.236907 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:40:30.237010 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 21:40:30.237156 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":1}
I20260926 21:40:30.238423 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.232348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39097 (local address 127.0.97.3:42599)
0926 21:40:30.233326 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:30.233344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:30.233481 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:30.233822 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:30.233826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:30.233850 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:30.234163 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.234172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.235137 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.235143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.236481 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:30.236492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.237768 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.237776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:30.237941 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.237981 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 21:40:30.238090 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 21:40:30.238173 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":2,"thread_start_us":568,"threads_started":1}
I20260926 21:40:30.238606 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.232387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42599 (local address 127.0.97.2:39097)
0926 21:40:30.232637 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.232646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.233437 (+ 791us) server_negotiation.cc:407] Connection header received
0926 21:40:30.233644 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.233648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.233705 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.233798 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:30.234316 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.234982 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.236644 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.237479 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.237643 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.237730 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.238155 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.238321 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 21:40:30.238683 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793ca6f46e274f42b2e0dbc69bde3934" candidate_uuid: "b4266a1190644edbb96dabe8d7869715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022373814611430b8100c190aa93364a" is_pre_election: true
I20260926 21:40:30.239306 16039 raft_consensus.cc:2454] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4266a1190644edbb96dabe8d7869715 in term 0.
I20260926 21:40:30.239406 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793ca6f46e274f42b2e0dbc69bde3934" candidate_uuid: "b4266a1190644edbb96dabe8d7869715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4344e04c0752453f98d824535198e76b" is_pre_election: true
I20260926 21:40:30.239930 16173 raft_consensus.cc:2454] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4266a1190644edbb96dabe8d7869715 in term 0.
I20260926 21:40:30.240178 16243 leader_election.cc:302] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022373814611430b8100c190aa93364a, b4266a1190644edbb96dabe8d7869715; no voters:
I20260926 21:40:30.240590 16379 raft_consensus.cc:2781] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:40:30.240707 16379 raft_consensus.cc:482] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:40:30.240841 16379 raft_consensus.cc:3037] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:30.243130 16379 raft_consensus.cc:504] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } }
I20260926 21:40:30.243734 16379 leader_election.cc:288] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [CANDIDATE]: Term 1 election: Requested vote from peers 022373814611430b8100c190aa93364a (127.0.97.1:39055), 4344e04c0752453f98d824535198e76b (127.0.97.2:39097)
I20260926 21:40:30.244254 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793ca6f46e274f42b2e0dbc69bde3934" candidate_uuid: "b4266a1190644edbb96dabe8d7869715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4344e04c0752453f98d824535198e76b"
I20260926 21:40:30.244230 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793ca6f46e274f42b2e0dbc69bde3934" candidate_uuid: "b4266a1190644edbb96dabe8d7869715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022373814611430b8100c190aa93364a"
I20260926 21:40:30.244452 16173 raft_consensus.cc:3037] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:30.244453 16039 raft_consensus.cc:3037] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:30.247720 16173 raft_consensus.cc:2454] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4266a1190644edbb96dabe8d7869715 in term 1.
I20260926 21:40:30.247720 16039 raft_consensus.cc:2454] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4266a1190644edbb96dabe8d7869715 in term 1.
I20260926 21:40:30.248157 16240 leader_election.cc:302] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4344e04c0752453f98d824535198e76b, b4266a1190644edbb96dabe8d7869715; no voters:
I20260926 21:40:30.248505 16379 raft_consensus.cc:2781] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 1 FOLLOWER]: Leader election won for term 1
W20260926 21:40:30.248643 16085 tablet.cc:2369] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:40:30.248994 16379 raft_consensus.cc:686] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 1 LEADER]: Becoming Leader. State: Replica: b4266a1190644edbb96dabe8d7869715, State: Running, Role: LEADER
I20260926 21:40:30.249547 16379 consensus_queue.cc:237] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } }
I20260926 21:40:30.254384 15895 catalog_manager.cc:5495] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4266a1190644edbb96dabe8d7869715 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b4266a1190644edbb96dabe8d7869715" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } health_report { overall_health: HEALTHY } } }
W20260926 21:40:30.278021 16219 tablet.cc:2369] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:40:30.290624 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.284765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.0.97.3:39485)
0926 21:40:30.284970 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.284974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.285072 (+ 98us) server_negotiation.cc:407] Connection header received
0926 21:40:30.285356 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.285359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.285485 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.285625 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:40:30.286019 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.286864 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.288428 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.288819 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.288914 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.289505 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:40:30.290013 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:40:30.290140 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.290257 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.290325 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:40:30.298683 16173 raft_consensus.cc:1261] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [term 1 FOLLOWER]: Refusing update from remote peer b4266a1190644edbb96dabe8d7869715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:40:30.298686 16039 raft_consensus.cc:1261] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [term 1 FOLLOWER]: Refusing update from remote peer b4266a1190644edbb96dabe8d7869715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:40:30.299787 16381 consensus_queue.cc:1035] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:40:30.300149 16379 consensus_queue.cc:1035] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 21:40:30.302500 16353 tablet.cc:2369] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:40:30.313978 16385 mvcc.cc:205] Tried to move back new op lower bound from 7333719368892956672 to 7333719368706166784. Current Snapshot: MvccSnapshot[applied={T|T < 7333719368892956672}]
I20260926 21:40:30.326649 15895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:40:30.326903 15895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:40:30.330375 15895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Sending DeleteTablet for 3 replicas of tablet 793ca6f46e274f42b2e0dbc69bde3934
I20260926 21:40:30.331526 16019 tablet_service.cc:1514] Processing DeleteTablet for tablet 793ca6f46e274f42b2e0dbc69bde3934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38818
I20260926 21:40:30.331526 16153 tablet_service.cc:1514] Processing DeleteTablet for tablet 793ca6f46e274f42b2e0dbc69bde3934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60672
I20260926 21:40:30.332249 16287 tablet_service.cc:1514] Processing DeleteTablet for tablet 793ca6f46e274f42b2e0dbc69bde3934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56670
I20260926 21:40:30.333137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 15956
I20260926 21:40:30.342397 16396 tablet_replica.cc:330] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: stopping tablet replica
I20260926 21:40:30.343009 16396 raft_consensus.cc:2229] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:40:30.343809 16396 raft_consensus.cc:2258] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:30.346231 16396 ts_tablet_manager.cc:1905] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:30.353718 16394 tablet_replica.cc:330] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: stopping tablet replica
I20260926 21:40:30.354197 16394 raft_consensus.cc:2229] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:40:30.354744 16394 raft_consensus.cc:2258] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:30.354883 16396 ts_tablet_manager.cc:1918] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:30.355067 16396 log.cc:1196] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/793ca6f46e274f42b2e0dbc69bde3934
I20260926 21:40:30.355624 16395 tablet_replica.cc:330] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: stopping tablet replica
I20260926 21:40:30.355667 16396 ts_tablet_manager.cc:1939] T 793ca6f46e274f42b2e0dbc69bde3934 P b4266a1190644edbb96dabe8d7869715: Deleting consensus metadata
I20260926 21:40:30.356056 16395 raft_consensus.cc:2229] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:40:30.356608 16395 raft_consensus.cc:2258] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:30.358070 15880 catalog_manager.cc:4841] TS b4266a1190644edbb96dabe8d7869715 (127.0.97.3:39485): tablet 793ca6f46e274f42b2e0dbc69bde3934 (table test-table [id=8330d5252a2f4e38a9b6b1f246a86a39]) successfully deleted
I20260926 21:40:30.358788 16395 ts_tablet_manager.cc:1905] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 21:40:30 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:40:30.362165 16394 ts_tablet_manager.cc:1905] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:30.366016 16395 ts_tablet_manager.cc:1918] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:30.366176 16395 log.cc:1196] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/793ca6f46e274f42b2e0dbc69bde3934
I20260926 21:40:30.366734 16395 ts_tablet_manager.cc:1939] T 793ca6f46e274f42b2e0dbc69bde3934 P 022373814611430b8100c190aa93364a: Deleting consensus metadata
I20260926 21:40:30.368729 15882 catalog_manager.cc:4841] TS 022373814611430b8100c190aa93364a (127.0.97.1:39055): tablet 793ca6f46e274f42b2e0dbc69bde3934 (table test-table [id=8330d5252a2f4e38a9b6b1f246a86a39]) successfully deleted
I20260926 21:40:30.369710 16394 ts_tablet_manager.cc:1918] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:30.369879 16394 log.cc:1196] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/793ca6f46e274f42b2e0dbc69bde3934
I20260926 21:40:30.370095 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.334077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.0.97.1:39055)
0926 21:40:30.334298 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.334307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.342461 (+ 8154us) server_negotiation.cc:407] Connection header received
0926 21:40:30.343614 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.343635 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.343758 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.343854 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:30.348085 (+ 4231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.349296 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.355552 (+ 6256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.356294 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.356434 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.362272 (+ 5838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:30.362289 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:30.362309 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:30.362361 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:30.366465 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.367429 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.367437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.367441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.367610 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.368095 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.368100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.368103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.368552 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:30.368831 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.369470 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.369727 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:40:30.370406 16394 ts_tablet_manager.cc:1939] T 793ca6f46e274f42b2e0dbc69bde3934 P 4344e04c0752453f98d824535198e76b: Deleting consensus metadata
I20260926 21:40:30.372253 15879 catalog_manager.cc:4841] TS 4344e04c0752453f98d824535198e76b (127.0.97.2:39097): tablet 793ca6f46e274f42b2e0dbc69bde3934 (table test-table [id=8330d5252a2f4e38a9b6b1f246a86a39]) successfully deleted
I20260926 21:40:30.372779 16079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:30.471714 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 15956
I20260926 21:40:30.489748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 16090
Sep 26 21:40:30 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:40:30.507627 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.490664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.0.97.2:39097)
0926 21:40:30.490867 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.490873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.490895 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:40:30.491034 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.491040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.491135 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.491254 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:30.492725 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.493878 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.495220 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.495909 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.496040 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.500646 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:30.500671 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:30.500692 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:30.500738 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:30.504636 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.505508 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.505518 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.505522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.505685 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.506024 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.506029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.506033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.506501 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:30.506808 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.507022 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.507249 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:40:30.508427 16213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:30.631705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 16090
I20260926 21:40:30.649125 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 16224
Sep 26 21:40:30 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:40:30.668367 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:30.649981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.0.97.3:39485)
0926 21:40:30.650193 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:30.650199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:30.650221 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:40:30.650378 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:30.650385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:30.650483 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:30.650617 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:30.651959 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.653388 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.654768 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:30.655460 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:30.655607 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:30.660617 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:30.660637 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:30.660661 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:30.660723 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:30.664856 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.665781 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.665792 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.665796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.665991 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:30.666561 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:30.666566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:30.666571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:30.667064 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:30.667396 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:30.667718 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 21:40:30.667957 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:40:30.669275 16347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:30.793505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 16224
I20260926 21:40:30.813099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 15863
I20260926 21:40:30.814416 15925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:30.926163 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 15863
I20260926 21:40:50.948465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--webserver_interface=127.0.97.62
--webserver_port=45271
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:40:51.279026 16402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:51.279325 16402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:51.279415 16402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:51.286913 16402 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 21:40:51.287016 16402 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 21:40:51.287055 16402 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 21:40:51.287098 16402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:40:51.287158 16402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:51.287218 16402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:51.287262 16402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:40:51.287308 16402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:40:51.297324 16402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:51.298563 16402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:51.299820 16402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:51.322958 16408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:51.323004 16411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:51.322978 16409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:51.323956 16402 server_base.cc:1034] running on GCE node
I20260926 21:40:51.324878 16402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:51.326692 16402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:51.327972 16402 hybrid_clock.cc:648] HybridClock initialized: now 1790458851327895 us; error 42 us; skew 500 ppm
I20260926 21:40:51.328418 16402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:51 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:51.332300 16402 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:40:51.334040 16402 webserver.cc:460] Webserver started at http://127.0.97.62:45271/ using document root <none> and password file <none>
I20260926 21:40:51.334887 16402 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:51.335037 16402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:51.341252 16402 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 21:40:51.344230 16418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:51.345752 16402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:40:51.345945 16402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "0f105a9cf0e649a1a9dff8026d044565"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "d868388eb411901894d5df7384f522c2"
server_key_iv: "6936e7d0193938814bdb48547dd83fe8"
server_key_version: "encryptionkey@0"
I20260926 21:40:51.347026 16402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:51.373904 16402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:51.378386 16402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:51.378702 16402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:51.398504 16402 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42309
I20260926 21:40:51.398535 16469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42309 every 8 connection(s)
I20260926 21:40:51.400185 16402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:40:51.402405 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 16402
I20260926 21:40:51.403265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39055
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42309
--webserver_port=40035
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:40:51.407840 16470 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:40:51.411412 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Bootstrap starting.
I20260926 21:40:51.431756 16470 log.cc:824] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:51.444962 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:40:51.445694 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Bootstrap complete.
I20260926 21:40:51.456007 16470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:40:51.456987 16470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f105a9cf0e649a1a9dff8026d044565, State: Initialized, Role: FOLLOWER
I20260926 21:40:51.457707 16470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:40:51.457999 16470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:40:51.458175 16470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:40:51.458382 16470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 1 FOLLOWER]: Advancing to term 2
I20260926 21:40:51.462627 16470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:40:51.463224 16470 leader_election.cc:302] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f105a9cf0e649a1a9dff8026d044565; no voters:
I20260926 21:40:51.463980 16470 leader_election.cc:288] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 21:40:51.464136 16474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 21:40:51.465456 16474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 2 LEADER]: Becoming Leader. State: Replica: 0f105a9cf0e649a1a9dff8026d044565, State: Running, Role: LEADER
I20260926 21:40:51.466147 16474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:40:51.466535 16470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:40:51.471731 16476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f105a9cf0e649a1a9dff8026d044565. Latest consensus state: current_term: 2 leader_uuid: "0f105a9cf0e649a1a9dff8026d044565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260926 21:40:51.472224 16476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:51.471705 16475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0f105a9cf0e649a1a9dff8026d044565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260926 21:40:51.472980 16475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: This master's current role is: LEADER
I20260926 21:40:51.473841 16484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:40:51.482576 16484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:40:51.484624 16484 catalog_manager.cc:1259] Loaded cluster ID: 997f8bc0e4f34d32879fcbb7afeba986
I20260926 21:40:51.484875 16484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:40:51.490666 16484 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:40:51.493134 16484 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Loaded TSK: 0
I20260926 21:40:51.516029 16484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Generated new TSK 1
I20260926 21:40:51.516443 16484 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 21:40:51.763355 16472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:51.763680 16472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:51.763837 16472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:51.771791 16472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:51.771958 16472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:51.772146 16472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:40:51.782907 16472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40035
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:51.784476 16472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:51.786093 16472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:51.801926 16498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:51.802453 16500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:51.802631 16497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:51.803212 16472 server_base.cc:1034] running on GCE node
I20260926 21:40:51.803965 16472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:51.805014 16472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:51.806255 16472 hybrid_clock.cc:648] HybridClock initialized: now 1790458851806183 us; error 67 us; skew 500 ppm
I20260926 21:40:51.806614 16472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:51 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:51.810220 16472 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:40:51.812103 16472 webserver.cc:460] Webserver started at http://127.0.97.1:40035/ using document root <none> and password file <none>
I20260926 21:40:51.812902 16472 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:51.813071 16472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:51.819260 16472 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:40:51.822342 16507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:51.823887 16472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:40:51.824072 16472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "022373814611430b8100c190aa93364a"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "7b20244ab197311986371ca44cab5980"
server_key_iv: "84d53bfe57911225786e4e7994e597e7"
server_key_version: "encryptionkey@0"
I20260926 21:40:51.825110 16472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:51.871311 16472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:51.875960 16472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:51.876250 16472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:51.877344 16472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:51.878746 16472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:51.878855 16472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:51.878983 16472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:51.879076 16472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:51.912405 16472 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39055
I20260926 21:40:51.912496 16619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39055 every 8 connection(s)
I20260926 21:40:51.913991 16472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:40:51.918334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 16472
I20260926 21:40:51.919006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39097
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42309
--webserver_port=37731
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:40:51 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:51.940338 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:51.916945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46411 (local address 127.0.97.62:42309)
0926 21:40:51.917656 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:51.917679 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:51.918285 (+ 606us) server_negotiation.cc:407] Connection header received
0926 21:40:51.920666 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:51.920680 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:51.921060 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:51.921494 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:51.923119 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:51.924386 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:51.926773 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:51.927483 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:51.927698 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:51.931903 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:51.931915 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:51.931933 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:51.931990 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:51.935465 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:51.936369 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:51.936377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:51.936383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:51.936558 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:51.937028 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:51.937038 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:51.937040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:51.937588 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:51.937806 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:51.938598 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 21:40:51.939000 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":303,"threads_started":1}
I20260926 21:40:51.940335 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:51.917124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:46411)
0926 21:40:51.918104 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:51.918148 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:51.920062 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:51.921661 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:51.921672 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:51.922333 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:51.922899 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:51.922918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:51.924576 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:51.924580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:51.926595 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:51.926617 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:51.927737 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:51.927745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:51.928030 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:51.928775 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:51.928800 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:51.931514 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:51.935668 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:51.935680 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:51.935704 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:51.936197 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:51.936692 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:51.936697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:51.936701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:51.936902 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:51.937733 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:51.937741 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:51.938101 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 21:40:51.938371 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 21:40:51.938977 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":455,"threads_started":1}
I20260926 21:40:51.943679 16620 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260926 21:40:51.943991 16620 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:51.944725 16620 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260926 21:40:51.947165 16435 ts_manager.cc:194] Registered new tserver with Master: 022373814611430b8100c190aa93364a (127.0.97.1:39055)
I20260926 21:40:51.949353 16435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46411
W20260926 21:40:52.255532 16624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:52.255841 16624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:52.255936 16624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:52.263273 16624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:52.263397 16624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:52.263514 16624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:40:52.274067 16624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37731
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:52.275388 16624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:52.276639 16624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:52.290125 16629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:52.290208 16630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:52.290175 16632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:52.290988 16624 server_base.cc:1034] running on GCE node
I20260926 21:40:52.291610 16624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:52.292569 16624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:52.293807 16624 hybrid_clock.cc:648] HybridClock initialized: now 1790458852293741 us; error 55 us; skew 500 ppm
I20260926 21:40:52.294124 16624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:52 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:52.297358 16624 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:40:52.298866 16624 webserver.cc:460] Webserver started at http://127.0.97.2:37731/ using document root <none> and password file <none>
I20260926 21:40:52.299541 16624 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:52.299654 16624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:52.305029 16624 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:40:52.307900 16639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:52.309290 16624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:40:52.309453 16624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "4344e04c0752453f98d824535198e76b"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "0db0e9c94b4bca5e5cdfd96c6cdb996e"
server_key_iv: "cb0aace4191034b031856059e53e8632"
server_key_version: "encryptionkey@0"
I20260926 21:40:52.310426 16624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:52.323542 16624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:52.327524 16624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:52.327800 16624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:52.328830 16624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:52.330271 16624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:52.330356 16624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:52.330476 16624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:52.330585 16624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:52.363647 16624 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39097
I20260926 21:40:52.363752 16751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39097 every 8 connection(s)
I20260926 21:40:52.365144 16624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:40:52.373883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 16624
I20260926 21:40:52.374557 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39485
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42309
--webserver_port=43243
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 21:40:52 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:52.389262 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.367943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57951 (local address 127.0.97.62:42309)
0926 21:40:52.368131 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:52.368138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:52.369475 (+ 1337us) server_negotiation.cc:407] Connection header received
0926 21:40:52.371460 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:52.371465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:52.371540 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:52.371625 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:52.373505 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.374519 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.377305 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.377740 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.377856 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.382602 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:52.382608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:52.382613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:52.382679 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:52.385490 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:52.386598 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:52.386602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:52.386605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:52.386733 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:52.387349 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:52.387352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:52.387353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:52.387627 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:52.387763 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:52.388759 (+ 996us) server_negotiation.cc:299] Negotiation successful
0926 21:40:52.388940 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:40:52.391106 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.368465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:57951)
0926 21:40:52.369264 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:52.369310 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:52.371159 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:52.371864 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:52.371873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:52.372421 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:52.373294 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.373316 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.374701 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.374708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.377073 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.377092 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.377985 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.377994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.378354 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.379167 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:52.379204 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:52.382263 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:52.385662 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:52.385677 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:52.385699 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:52.386461 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:52.386847 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:52.386855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:52.386862 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:52.387224 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:52.387773 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:52.387784 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:52.388327 (+ 543us) client_negotiation.cc:769] Sending connection context
0926 21:40:52.388675 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 21:40:52.389394 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":289,"threads_started":1}
I20260926 21:40:52.392827 16752 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260926 21:40:52.393178 16752 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:52.393980 16752 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260926 21:40:52.395748 16435 ts_manager.cc:194] Registered new tserver with Master: 4344e04c0752453f98d824535198e76b (127.0.97.2:39097)
I20260926 21:40:52.397174 16435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57951
W20260926 21:40:52.715607 16755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:40:52.715894 16755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:40:52.715991 16755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:40:52.723436 16755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:40:52.723565 16755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:40:52.723683 16755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:40:52.735016 16755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43243
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:40:52.736337 16755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:40:52.737728 16755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:40:52.751158 16760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:52.751132 16761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:40:52.751682 16763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:40:52.752382 16755 server_base.cc:1034] running on GCE node
I20260926 21:40:52.753008 16755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:40:52.754043 16755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:40:52.755291 16755 hybrid_clock.cc:648] HybridClock initialized: now 1790458852755247 us; error 40 us; skew 500 ppm
I20260926 21:40:52.755622 16755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:40:52 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:40:52.758963 16755 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:40:52.760496 16755 webserver.cc:460] Webserver started at http://127.0.97.3:43243/ using document root <none> and password file <none>
I20260926 21:40:52.761276 16755 fs_manager.cc:359] Metadata directory not provided
I20260926 21:40:52.761430 16755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:40:52.767145 16755 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:40:52.770296 16770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:52.771819 16755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:40:52.772018 16755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "b4266a1190644edbb96dabe8d7869715"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "5b2c49cd5d03a713729e4be6437e7e6e"
server_key_iv: "0157526aa0e5d501b152ac78af96ae1c"
server_key_version: "encryptionkey@0"
I20260926 21:40:52.773100 16755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:40:52.790297 16755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:40:52.795241 16755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:40:52.795567 16755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:40:52.796633 16755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:40:52.798208 16755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:40:52.798300 16755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:52.798450 16755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:40:52.798544 16755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:40:52.832376 16755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39485
I20260926 21:40:52.832460 16882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39485 every 8 connection(s)
I20260926 21:40:52.833931 16755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:40:52.843355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 16755
Sep 26 21:40:52 dist-test-slave-lhhq krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:40:52.855336 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.836486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33085 (local address 127.0.97.62:42309)
0926 21:40:52.836675 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:52.836679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:52.837853 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 21:40:52.839727 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:52.839734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:52.839832 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:52.839950 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:52.841592 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.842353 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.844092 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.844752 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.844881 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.848789 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:52.848794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:52.848797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:52.848827 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:52.851675 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:52.852617 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:52.852620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:52.852623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:52.852752 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:52.853304 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:52.853307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:52.853309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:52.853583 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:52.853694 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:52.854675 (+ 981us) server_negotiation.cc:299] Negotiation successful
0926 21:40:52.854971 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:40:52.856400 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.836911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:33085)
0926 21:40:52.837682 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:52.837728 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:52.839436 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:52.840207 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:52.840216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:52.840759 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:40:52.841416 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.841437 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.842528 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.842532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.843918 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.843935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.845067 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.845074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.845368 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.846208 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:40:52.846234 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:40:52.848582 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:40:52.851822 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:52.851836 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:52.851856 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:52.852496 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:52.852863 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:40:52.852869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:40:52.852875 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:40:52.853149 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:40:52.853710 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:40:52.853718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:40:52.854221 (+ 503us) client_negotiation.cc:769] Sending connection context
0926 21:40:52.854534 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 21:40:52.854965 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":323,"threads_started":1}
I20260926 21:40:52.858107 16883 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42309
I20260926 21:40:52.858453 16883 heartbeater.cc:464] Registering TS with master...
I20260926 21:40:52.859385 16883 heartbeater.cc:511] Master 127.0.97.62:42309 requested a full tablet report, sending...
I20260926 21:40:52.860626 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.844288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.0.97.62:42309)
0926 21:40:52.844876 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:52.844881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:52.845234 (+ 353us) server_negotiation.cc:407] Connection header received
0926 21:40:52.845421 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:52.845425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:52.845509 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:52.845627 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:52.847290 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.848074 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.849970 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.850683 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.850849 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.854418 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:52.854423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:52.854426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:52.854452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:52.857585 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:52.858491 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:52.858498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:52.858501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:52.858637 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:52.858979 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:52.858984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:52.858987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:52.859361 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:52.859621 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:52.860030 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 21:40:52.860260 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":312,"threads_started":1}
I20260926 21:40:52.861096 16435 ts_manager.cc:194] Registered new tserver with Master: b4266a1190644edbb96dabe8d7869715 (127.0.97.3:39485)
I20260926 21:40:52.862085 16435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33085
I20260926 21:40:52.863288 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:40:52.870894 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.864322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.0.97.62:42309)
0926 21:40:52.864495 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:52.864501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:52.865476 (+ 975us) server_negotiation.cc:407] Connection header received
0926 21:40:52.865709 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:52.865713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:52.865831 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:52.865973 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:40:52.866472 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.867237 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.868838 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.869351 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.869471 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.869955 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:40:52.870147 (+ 192us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 21:40:52.870611 (+ 464us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56212: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 21:40:52.871096 16886 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56212: token expired
W20260926 21:40:52.871831 16359 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:42309)
I20260926 21:40:52.886472 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.872360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.0.97.62:42309)
0926 21:40:52.872508 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:52.872512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:52.872558 (+ 46us) server_negotiation.cc:407] Connection header received
0926 21:40:52.872730 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:52.872734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:52.872788 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:52.872864 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:52.874787 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.875489 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.877644 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.878147 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.878281 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.881024 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:52.881040 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:52.881043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:52.881078 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:52.883976 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:52.884662 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:52.884666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:52.884668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:52.884789 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:52.885120 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:52.885123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:52.885125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:52.885498 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:52.885678 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:52.885982 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 21:40:52.886154 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:40:52.890139 16434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:40:52.897492 16434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:40:52.920297 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.909370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39485 (local address 127.0.0.1:36160)
0926 21:40:52.910925 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:52.910982 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:52.911208 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:52.912336 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:52.912343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:52.912372 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:52.912764 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.912779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.914850 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.914857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.917134 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.917159 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.919324 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.919331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.919515 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.919567 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 21:40:52.919696 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 21:40:52.919895 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"thread_start_us":1172,"threads_started":1}
I20260926 21:40:52.921564 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.909664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.0.97.3:39485)
0926 21:40:52.910500 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:52.910511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:52.911425 (+ 914us) server_negotiation.cc:407] Connection header received
0926 21:40:52.911546 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:52.911552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:52.911825 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:52.912035 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:52.912980 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.914661 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.917346 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.918799 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.918920 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.919167 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:52.920987 (+ 1820us) server_negotiation.cc:299] Negotiation successful
0926 21:40:52.921216 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":521,"threads_started":1}
I20260926 21:40:52.926467 16818 tablet_service.cc:1467] Processing CreateTablet for tablet adfe97ea79174d4888a217475cffc24c (DEFAULT_TABLE table=test-table [id=dc3ab963dfa84c168faa679259c31f10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:52.928133 16818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfe97ea79174d4888a217475cffc24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:52.931329 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.910628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39055 (local address 127.0.0.1:34256)
0926 21:40:52.921908 (+ 11280us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:52.921923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:52.922020 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:52.924338 (+ 2318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:52.924345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:52.924377 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:52.924575 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.924589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.926740 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.926749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.928181 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.928222 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.930404 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.930413 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.930619 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.930659 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 21:40:52.930743 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 21:40:52.930930 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11157,"thread_start_us":814,"threads_started":1}
I20260926 21:40:52.931759 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.909370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39097 (local address 127.0.0.1:46020)
0926 21:40:52.911314 (+ 1944us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:52.911338 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:52.911434 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:52.923775 (+ 12341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:52.923781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:52.923814 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:52.924201 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.924220 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.926336 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.926342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.928432 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:52.928452 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.930875 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.930882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:52.931087 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.931124 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 21:40:52.931227 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 21:40:52.931390 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":12429,"threads_started":1}
I20260926 21:40:52.931854 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.909802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.0.97.2:39097)
0926 21:40:52.923045 (+ 13243us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:52.923054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:52.923104 (+ 50us) server_negotiation.cc:407] Connection header received
0926 21:40:52.923224 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:52.923232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:52.923534 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:52.923770 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:52.924467 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.926164 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.928666 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.930319 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.930454 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.930696 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:52.931306 (+ 610us) server_negotiation.cc:299] Negotiation successful
0926 21:40:52.931531 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12994,"spinlock_wait_cycles":21760,"thread_start_us":12955,"threads_started":1}
I20260926 21:40:52.933130 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:52.911261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.0.97.1:39055)
0926 21:40:52.923605 (+ 12344us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:52.923612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:52.923638 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:40:52.923757 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:52.923765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:52.924048 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:52.924769 (+ 721us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:52.925059 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.926594 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.928418 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:52.930180 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:52.932206 (+ 2026us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:52.932468 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:52.932544 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 21:40:52.932721 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12177,"thread_start_us":12026,"threads_started":1}
I20260926 21:40:52.936072 16687 tablet_service.cc:1467] Processing CreateTablet for tablet adfe97ea79174d4888a217475cffc24c (DEFAULT_TABLE table=test-table [id=dc3ab963dfa84c168faa679259c31f10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:52.936193 16555 tablet_service.cc:1467] Processing CreateTablet for tablet adfe97ea79174d4888a217475cffc24c (DEFAULT_TABLE table=test-table [id=dc3ab963dfa84c168faa679259c31f10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:40:52.937682 16687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfe97ea79174d4888a217475cffc24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:52.937692 16555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfe97ea79174d4888a217475cffc24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:40:52.945588 16899 tablet_bootstrap.cc:492] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: Bootstrap starting.
I20260926 21:40:52.949784 16899 tablet_bootstrap.cc:654] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:52.951207 16899 log.cc:824] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:52.951793 16900 tablet_bootstrap.cc:492] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: Bootstrap starting.
I20260926 21:40:52.952932 16620 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
I20260926 21:40:52.955688 16900 tablet_bootstrap.cc:654] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:52.955684 16899 tablet_bootstrap.cc:492] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: No bootstrap required, opened a new log
I20260926 21:40:52.956079 16899 ts_tablet_manager.cc:1397] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 21:40:52.956878 16900 log.cc:824] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:52.957994 16902 tablet_bootstrap.cc:492] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: Bootstrap starting.
I20260926 21:40:52.961663 16902 tablet_bootstrap.cc:654] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: Neither blocks nor log segments found. Creating new log.
I20260926 21:40:52.961736 16900 tablet_bootstrap.cc:492] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: No bootstrap required, opened a new log
I20260926 21:40:52.962021 16900 ts_tablet_manager.cc:1397] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 21:40:52.962975 16902 log.cc:824] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:40:52.968506 16899 raft_consensus.cc:348] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } }
I20260926 21:40:52.969507 16899 raft_consensus.cc:374] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:52.969702 16899 raft_consensus.cc:729] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4266a1190644edbb96dabe8d7869715, State: Initialized, Role: FOLLOWER
I20260926 21:40:52.971125 16899 consensus_queue.cc:260] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } }
I20260926 21:40:52.976529 16900 raft_consensus.cc:348] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } }
I20260926 21:40:52.976940 16883 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
I20260926 21:40:52.977052 16902 tablet_bootstrap.cc:492] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: No bootstrap required, opened a new log
I20260926 21:40:52.977079 16900 raft_consensus.cc:374] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:52.977274 16900 raft_consensus.cc:729] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4344e04c0752453f98d824535198e76b, State: Initialized, Role: FOLLOWER
I20260926 21:40:52.977421 16902 ts_tablet_manager.cc:1397] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20260926 21:40:52.978080 16899 ts_tablet_manager.cc:1428] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: Time spent starting tablet: real 0.022s user 0.018s sys 0.000s
I20260926 21:40:52.977993 16900 consensus_queue.cc:260] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } }
I20260926 21:40:52.984048 16752 heartbeater.cc:503] Master 127.0.97.62:42309 was elected leader, sending a full tablet report...
I20260926 21:40:52.984789 16900 ts_tablet_manager.cc:1428] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: Time spent starting tablet: real 0.023s user 0.014s sys 0.008s
I20260926 21:40:52.988013 16902 raft_consensus.cc:348] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } }
I20260926 21:40:52.988549 16902 raft_consensus.cc:374] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:40:52.988678 16902 raft_consensus.cc:729] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022373814611430b8100c190aa93364a, State: Initialized, Role: FOLLOWER
I20260926 21:40:52.989383 16902 consensus_queue.cc:260] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } }
I20260926 21:40:52.991074 16902 ts_tablet_manager.cc:1428] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260926 21:40:53.085559 16884 tablet.cc:2369] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:40:53.117367 16753 tablet.cc:2369] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:40:53.166486 16621 tablet.cc:2369] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:40:53.216449 16906 raft_consensus.cc:482] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:40:53.216800 16906 raft_consensus.cc:504] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } }
I20260926 21:40:53.218549 16906 leader_election.cc:288] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4266a1190644edbb96dabe8d7869715 (127.0.97.3:39485), 022373814611430b8100c190aa93364a (127.0.97.1:39055)
I20260926 21:40:53.225601 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:53.218880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39055 (local address 127.0.97.2:35395)
0926 21:40:53.219859 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:53.219873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:53.220054 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:53.220385 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:53.220390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:53.220415 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:53.220826 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:53.220834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.221897 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.221904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:53.223491 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:53.223503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.224674 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.224680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:53.224851 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:53.224875 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 21:40:53.224982 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 21:40:53.225121 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":265,"spinlock_wait_cycles":57088,"thread_start_us":504,"threads_started":1}
I20260926 21:40:53.225848 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:53.218881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34665 (local address 127.0.97.3:39485)
0926 21:40:53.219070 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:53.219075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:53.219956 (+ 881us) server_negotiation.cc:407] Connection header received
0926 21:40:53.220210 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:53.220216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:53.220292 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:53.220393 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:53.221020 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.221882 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.223793 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.224678 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.224837 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:53.224954 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:53.225311 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 21:40:53.225581 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 21:40:53.225917 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:53.218876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39485 (local address 127.0.97.2:34665)
0926 21:40:53.219829 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0926 21:40:53.219850 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:40:53.220057 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:40:53.220406 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:40:53.220411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:40:53.220435 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:40:53.220874 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:53.220882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.222037 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.222043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:53.223628 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:40:53.223640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.224909 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.224916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:40:53.225106 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:53.225130 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 21:40:53.225287 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 21:40:53.225427 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":1,"thread_start_us":453,"threads_started":1}
I20260926 21:40:53.225435 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:53.219146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35395 (local address 127.0.97.1:39055)
0926 21:40:53.219395 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:53.219401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:53.219974 (+ 573us) server_negotiation.cc:407] Connection header received
0926 21:40:53.220206 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:53.220210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:53.220269 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:53.220360 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:40:53.220981 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.221757 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.223646 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.224445 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.224602 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:53.224712 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:53.225009 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 21:40:53.225140 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 21:40:53.227164 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfe97ea79174d4888a217475cffc24c" candidate_uuid: "4344e04c0752453f98d824535198e76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4266a1190644edbb96dabe8d7869715" is_pre_election: true
I20260926 21:40:53.227394 16575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfe97ea79174d4888a217475cffc24c" candidate_uuid: "4344e04c0752453f98d824535198e76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022373814611430b8100c190aa93364a" is_pre_election: true
I20260926 21:40:53.227672 16838 raft_consensus.cc:2454] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4344e04c0752453f98d824535198e76b in term 0.
I20260926 21:40:53.227944 16575 raft_consensus.cc:2454] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4344e04c0752453f98d824535198e76b in term 0.
I20260926 21:40:53.228577 16641 leader_election.cc:302] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 022373814611430b8100c190aa93364a, 4344e04c0752453f98d824535198e76b, b4266a1190644edbb96dabe8d7869715; no voters:
I20260926 21:40:53.229120 16906 raft_consensus.cc:2781] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:40:53.229337 16906 raft_consensus.cc:482] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:40:53.229471 16906 raft_consensus.cc:3037] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:53.231758 16906 raft_consensus.cc:504] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } }
I20260926 21:40:53.232395 16906 leader_election.cc:288] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [CANDIDATE]: Term 1 election: Requested vote from peers b4266a1190644edbb96dabe8d7869715 (127.0.97.3:39485), 022373814611430b8100c190aa93364a (127.0.97.1:39055)
I20260926 21:40:53.232957 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfe97ea79174d4888a217475cffc24c" candidate_uuid: "4344e04c0752453f98d824535198e76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4266a1190644edbb96dabe8d7869715"
I20260926 21:40:53.232964 16575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfe97ea79174d4888a217475cffc24c" candidate_uuid: "4344e04c0752453f98d824535198e76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022373814611430b8100c190aa93364a"
I20260926 21:40:53.233167 16838 raft_consensus.cc:3037] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:53.233239 16575 raft_consensus.cc:3037] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:40:53.235394 16575 raft_consensus.cc:2454] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4344e04c0752453f98d824535198e76b in term 1.
I20260926 21:40:53.235406 16838 raft_consensus.cc:2454] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4344e04c0752453f98d824535198e76b in term 1.
I20260926 21:40:53.235879 16643 leader_election.cc:302] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 022373814611430b8100c190aa93364a, 4344e04c0752453f98d824535198e76b, b4266a1190644edbb96dabe8d7869715; no voters:
I20260926 21:40:53.236228 16906 raft_consensus.cc:2781] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:40:53.236680 16906 raft_consensus.cc:686] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 1 LEADER]: Becoming Leader. State: Replica: 4344e04c0752453f98d824535198e76b, State: Running, Role: LEADER
I20260926 21:40:53.237290 16906 consensus_queue.cc:237] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } }
I20260926 21:40:53.241854 16435 catalog_manager.cc:5495] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4344e04c0752453f98d824535198e76b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4344e04c0752453f98d824535198e76b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4344e04c0752453f98d824535198e76b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 } health_report { overall_health: UNKNOWN } } }
I20260926 21:40:53.300511 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:53.295134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.0.97.2:39097)
0926 21:40:53.295348 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:53.295353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:53.295431 (+ 78us) server_negotiation.cc:407] Connection header received
0926 21:40:53.295608 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:53.295612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:53.295695 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:53.295838 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:40:53.296174 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.296885 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.298373 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.298774 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.298867 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:53.299441 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:40:53.299945 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:40:53.300094 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:53.300221 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 21:40:53.300290 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 21:40:53.309340 16575 raft_consensus.cc:1261] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [term 1 FOLLOWER]: Refusing update from remote peer 4344e04c0752453f98d824535198e76b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:40:53.309341 16838 raft_consensus.cc:1261] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [term 1 FOLLOWER]: Refusing update from remote peer 4344e04c0752453f98d824535198e76b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:40:53.310412 16910 consensus_queue.cc:1035] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4266a1190644edbb96dabe8d7869715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:40:53.310758 16906 consensus_queue.cc:1035] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [LEADER]: Connected to new peer: Peer: permanent_uuid: "022373814611430b8100c190aa93364a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:40:53.328457 16915 mvcc.cc:205] Tried to move back new op lower bound from 7333719463144448000 to 7333719462863724544. Current Snapshot: MvccSnapshot[applied={T|T < 7333719463144448000}]
I20260926 21:40:53.340628 16434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:40:53.340915 16434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:40:53.344416 16434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Sending DeleteTablet for 3 replicas of tablet adfe97ea79174d4888a217475cffc24c
I20260926 21:40:53.345762 16818 tablet_service.cc:1514] Processing DeleteTablet for tablet adfe97ea79174d4888a217475cffc24c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36160
I20260926 21:40:53.345731 16687 tablet_service.cc:1514] Processing DeleteTablet for tablet adfe97ea79174d4888a217475cffc24c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46020
I20260926 21:40:53.346784 16924 tablet_replica.cc:330] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: stopping tablet replica
I20260926 21:40:53.346854 16555 tablet_service.cc:1514] Processing DeleteTablet for tablet adfe97ea79174d4888a217475cffc24c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:40:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34256
I20260926 21:40:53.347424 16924 raft_consensus.cc:2229] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:40:53.347869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 16472
I20260926 21:40:53.348289 16924 raft_consensus.cc:2258] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:53.349086 16923 tablet_replica.cc:330] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: stopping tablet replica
I20260926 21:40:53.349566 16923 raft_consensus.cc:2229] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:40:53.352012 16923 raft_consensus.cc:2258] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:53.353645 16924 ts_tablet_manager.cc:1905] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:53.358937 16925 tablet_replica.cc:330] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: stopping tablet replica
I20260926 21:40:53.359623 16925 raft_consensus.cc:2229] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:40:53.360278 16925 raft_consensus.cc:2258] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:40:53.362543 16923 ts_tablet_manager.cc:1905] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:53.362773 16925 ts_tablet_manager.cc:1905] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:40:53.369972 16923 ts_tablet_manager.cc:1918] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:53.370152 16923 log.cc:1196] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/adfe97ea79174d4888a217475cffc24c
I20260926 21:40:53.370368 16925 ts_tablet_manager.cc:1918] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:53.370651 16925 log.cc:1196] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/adfe97ea79174d4888a217475cffc24c
I20260926 21:40:53.370668 16923 ts_tablet_manager.cc:1939] T adfe97ea79174d4888a217475cffc24c P b4266a1190644edbb96dabe8d7869715: Deleting consensus metadata
I20260926 21:40:53.371264 16925 ts_tablet_manager.cc:1939] T adfe97ea79174d4888a217475cffc24c P 022373814611430b8100c190aa93364a: Deleting consensus metadata
I20260926 21:40:53.371987 16924 ts_tablet_manager.cc:1918] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:40:53.372218 16924 log.cc:1196] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/adfe97ea79174d4888a217475cffc24c
I20260926 21:40:53.372797 16924 ts_tablet_manager.cc:1939] T adfe97ea79174d4888a217475cffc24c P 4344e04c0752453f98d824535198e76b: Deleting consensus metadata
I20260926 21:40:53.373991 16422 catalog_manager.cc:4841] TS 022373814611430b8100c190aa93364a (127.0.97.1:39055): tablet adfe97ea79174d4888a217475cffc24c (table test-table [id=dc3ab963dfa84c168faa679259c31f10]) successfully deleted
I20260926 21:40:53.374764 16419 catalog_manager.cc:4841] TS 4344e04c0752453f98d824535198e76b (127.0.97.2:39097): tablet adfe97ea79174d4888a217475cffc24c (table test-table [id=dc3ab963dfa84c168faa679259c31f10]) successfully deleted
I20260926 21:40:53.375543 16420 catalog_manager.cc:4841] TS b4266a1190644edbb96dabe8d7869715 (127.0.97.3:39485): tablet adfe97ea79174d4888a217475cffc24c (table test-table [id=dc3ab963dfa84c168faa679259c31f10]) successfully deleted
I20260926 21:40:53.377337 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:53.348666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.0.97.1:39055)
0926 21:40:53.349326 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:53.349332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:53.349349 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:40:53.349408 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:53.349413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:53.349522 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:53.349630 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:53.351716 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.352906 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.354836 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.355510 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.355637 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:53.359758 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:53.359775 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:53.359795 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:53.359848 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:53.373100 (+ 13252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:53.374666 (+ 1566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:53.374675 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:53.374678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:53.374855 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:53.375369 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:53.375374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:53.375377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:53.375869 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:53.376248 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:53.376540 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 21:40:53.376772 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534}
I20260926 21:40:53.377962 16615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:53.489662 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 16472
I20260926 21:40:53.514621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 16624
I20260926 21:40:53.530594 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:53.515499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.0.97.2:39097)
0926 21:40:53.515699 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:53.515705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:53.515725 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:40:53.515790 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:53.515796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:53.515898 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:53.515999 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:53.517351 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.518549 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.519935 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.520616 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.520753 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:53.523412 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:53.523431 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:53.523452 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:53.523501 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:53.527519 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:53.528342 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:53.528352 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:53.528356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:53.528546 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:53.528892 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:53.528898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:53.528903 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:53.529390 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:53.529738 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:53.529930 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 21:40:53.530171 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:40:53.531442 16747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:53.649997 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 16624
I20260926 21:40:53.669636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 16755
I20260926 21:40:53.684530 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:40:53.670398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.0.97.3:39485)
0926 21:40:53.670600 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 21:40:53.670608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:40:53.670630 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:40:53.670838 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:40:53.670844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:40:53.670930 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:40:53.671036 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:40:53.672507 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.673348 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.674743 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:40:53.675415 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:40:53.675580 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:40:53.678487 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:40:53.678501 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:40:53.678521 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:40:53.678555 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:40:53.681594 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:53.682310 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:53.682320 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:53.682324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:53.682485 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:40:53.682840 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:40:53.682847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:40:53.682859 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:40:53.683281 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:40:53.683598 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 21:40:53.683935 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 21:40:53.684172 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:40:53.685225 16878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:53.778178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 16755
I20260926 21:40:53.795879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 16402
I20260926 21:40:53.797271 16465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:40:53.912297 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 16402
I20260926 21:41:13.934651 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--webserver_interface=127.0.97.62
--webserver_port=45271
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:14.271673 16930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:14.272058 16930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:14.272246 16930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:14.279651 16930 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 21:41:14.279811 16930 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 21:41:14.279902 16930 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 21:41:14.279996 16930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:14.280078 16930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:14.280207 16930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:14.280295 16930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:14.280380 16930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:14.290647 16930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:14.292042 16930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:14.293466 16930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:14.305274 16938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:14.305351 16935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:14.305568 16936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:14.306587 16930 server_base.cc:1034] running on GCE node
I20260926 21:41:14.307412 16930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:14.308866 16930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:14.310097 16930 hybrid_clock.cc:648] HybridClock initialized: now 1790458874310011 us; error 58 us; skew 500 ppm
I20260926 21:41:14.310392 16930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:14 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:14.322225 16930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:41:14.324158 16930 webserver.cc:460] Webserver started at http://127.0.97.62:45271/ using document root <none> and password file <none>
I20260926 21:41:14.324965 16930 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:14.325119 16930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:14.331363 16930 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:41:14.335433 16945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:14.336977 16930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:41:14.337252 16930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "0f105a9cf0e649a1a9dff8026d044565"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "d868388eb411901894d5df7384f522c2"
server_key_iv: "6936e7d0193938814bdb48547dd83fe8"
server_key_version: "encryptionkey@0"
I20260926 21:41:14.338372 16930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:14.356474 16930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:14.361397 16930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:14.361756 16930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:14.383122 16930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42309
I20260926 21:41:14.383217 16996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42309 every 8 connection(s)
I20260926 21:41:14.384963 16930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:14.389598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 16930
I20260926 21:41:14.390584 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39055
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42309
--webserver_port=40035
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:41:14.393792 16997 sys_catalog.cc:263] Verifying existing consensus state
I20260926 21:41:14.399358 16997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Bootstrap starting.
I20260926 21:41:14.420570 16997 log.cc:824] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:14.436439 16997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 21:41:14.437042 16997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Bootstrap complete.
I20260926 21:41:14.447073 16997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:41:14.448029 16997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f105a9cf0e649a1a9dff8026d044565, State: Initialized, Role: FOLLOWER
I20260926 21:41:14.448740 16997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:41:14.448999 16997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:14.449141 16997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:14.449361 16997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 2 FOLLOWER]: Advancing to term 3
I20260926 21:41:14.453313 16997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:41:14.453807 16997 leader_election.cc:302] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f105a9cf0e649a1a9dff8026d044565; no voters:
I20260926 21:41:14.454535 16997 leader_election.cc:288] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 21:41:14.454679 17001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 21:41:14.455415 17001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [term 3 LEADER]: Becoming Leader. State: Replica: 0f105a9cf0e649a1a9dff8026d044565, State: Running, Role: LEADER
I20260926 21:41:14.456009 17001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } }
I20260926 21:41:14.456640 16997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:14.458446 17002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f105a9cf0e649a1a9dff8026d044565. Latest consensus state: current_term: 3 leader_uuid: "0f105a9cf0e649a1a9dff8026d044565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260926 21:41:14.458776 17002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:14.459339 17003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0f105a9cf0e649a1a9dff8026d044565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f105a9cf0e649a1a9dff8026d044565" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42309 } } }
I20260926 21:41:14.459730 17003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:14.462417 17006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:14.471503 17006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:14.473074 17006 catalog_manager.cc:1259] Loaded cluster ID: 997f8bc0e4f34d32879fcbb7afeba986
I20260926 21:41:14.473176 17006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:14.477415 17006 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:14.479007 17006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Loaded TSK: 0
I20260926 21:41:14.479156 17006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f105a9cf0e649a1a9dff8026d044565: Loaded TSK: 1
F20260926 21:41:14.480299 17006 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7fc6b949ad google::LogMessage::Fail() at ??:0
@ 0x7f7fc6b9788a google::LogMessage::SendToLog() at ??:0
@ 0x7f7fc6b944de google::LogMessage::Flush() at ??:0
@ 0x7f7fc6b95a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7fca8f87d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7fca8e773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7fdc5c31c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7fdc5c2783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7fdc64e2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7fdc64e0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7fdc6985e9 std::function<>::operator()() at ??:0
@ 0x7f7fdc5c6cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7fdc5c4c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7fdc64c610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7fdc64c439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7fc97e0345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7fc97e391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7fc97e3799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7fc97beeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7fcca986db start_thread at ??:0
@ 0x7f7fc5a2e61f clone at ??:0
W20260926 21:41:14.780273 16999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:14.780572 16999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:14.780736 16999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:14.788410 16999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:14.788532 16999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:14.788715 16999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:41:14.798981 16999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40035
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:14.800364 16999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:14.801743 16999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:14.815268 17044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:14.815235 17047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:14.815272 17045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:14.816233 16999 server_base.cc:1034] running on GCE node
I20260926 21:41:14.816895 16999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:14.817983 16999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:14.819199 16999 hybrid_clock.cc:648] HybridClock initialized: now 1790458874819128 us; error 60 us; skew 500 ppm
I20260926 21:41:14.819574 16999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:14 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:14.822912 16999 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:41:14.824553 16999 webserver.cc:460] Webserver started at http://127.0.97.1:40035/ using document root <none> and password file <none>
I20260926 21:41:14.825320 16999 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:14.825475 16999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:14.831183 16999 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 21:41:14.834290 17054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:14.835753 16999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:41:14.835983 16999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "022373814611430b8100c190aa93364a"
format_stamp: "Formatted at 2026-09-26 21:40:28 on dist-test-slave-lhhq"
server_key: "7b20244ab197311986371ca44cab5980"
server_key_iv: "84d53bfe57911225786e4e7994e597e7"
server_key_version: "encryptionkey@0"
I20260926 21:41:14.837122 16999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:14.872556 16999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:14.876946 16999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:14.877279 16999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:14.878294 16999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:14.879961 16999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:14.880101 16999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:14.880236 16999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:14.880323 16999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:14.914856 16999 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39055
I20260926 21:41:14.915033 17166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39055 every 8 connection(s)
I20260926 21:41:14.916342 16999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:41:14.918865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 16999
I20260926 21:41:14.919621 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39097
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42309
--webserver_port=37731
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:41:14.921576 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:14.919646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:55033)
0926 21:41:14.920416 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:14.920650 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":310,"threads_started":1}
W20260926 21:41:14.922271 17167 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42309 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42309: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
I20260926 21:41:14.923070 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:14.922746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:47319)
0926 21:41:14.922893 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:14.922969 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 21:41:14.923834 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:14.923513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:51865)
0926 21:41:14.923651 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:14.923730 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 21:41:14.924153 17167 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 21:41:15.245947 17170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:15.246235 17170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:15.246330 17170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:15.254097 17170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:15.254204 17170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:15.254316 17170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:41:15.263967 17170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37731
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:15.265297 17170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:15.266582 17170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:15.279883 17178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:15.279914 17176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:15.280421 17175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:15.280632 17170 server_base.cc:1034] running on GCE node
I20260926 21:41:15.281271 17170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:15.282297 17170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:15.283553 17170 hybrid_clock.cc:648] HybridClock initialized: now 1790458875283492 us; error 50 us; skew 500 ppm
I20260926 21:41:15.283874 17170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:15 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:15.286916 17170 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:41:15.288347 17170 webserver.cc:460] Webserver started at http://127.0.97.2:37731/ using document root <none> and password file <none>
I20260926 21:41:15.289037 17170 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:15.289225 17170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:15.294602 17170 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 21:41:15.297402 17185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:15.298712 17170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:41:15.298888 17170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "4344e04c0752453f98d824535198e76b"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "0db0e9c94b4bca5e5cdfd96c6cdb996e"
server_key_iv: "cb0aace4191034b031856059e53e8632"
server_key_version: "encryptionkey@0"
I20260926 21:41:15.299894 17170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:15.323848 17170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:15.328044 17170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:15.328327 17170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:15.329366 17170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:15.330688 17170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:15.330807 17170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:15.330936 17170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:15.331040 17170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:15.365154 17170 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39097
I20260926 21:41:15.365304 17297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39097 every 8 connection(s)
I20260926 21:41:15.366688 17170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:41:15.371534 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:15.369642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:34275)
0926 21:41:15.370381 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:15.370604 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":294,"threads_started":1}
W20260926 21:41:15.372164 17298 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42309 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42309: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
I20260926 21:41:15.373019 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:15.372701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:35073)
0926 21:41:15.372827 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:15.372919 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 21:41:15.374051 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:15.373635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:44985)
0926 21:41:15.373814 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:15.373909 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 21:41:15.374028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 17170
W20260926 21:41:15.374284 17298 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 21:41:15.374689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39485
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42309
--webserver_port=43243
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:15.702446 17301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:15.702721 17301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:15.702903 17301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:15.710085 17301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:15.710214 17301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:15.710394 17301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:41:15.720177 17301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43243
--tserver_master_addrs=127.0.97.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:15.721977 17301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:15.723354 17301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:15.739471 17307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:15.739535 17306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:15.740087 17309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:15.740214 17301 server_base.cc:1034] running on GCE node
I20260926 21:41:15.740893 17301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:15.742059 17301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:15.743276 17301 hybrid_clock.cc:648] HybridClock initialized: now 1790458875743209 us; error 75 us; skew 500 ppm
I20260926 21:41:15.743657 17301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:15 dist-test-slave-lhhq krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:15.747370 17301 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:41:15.749318 17301 webserver.cc:460] Webserver started at http://127.0.97.3:43243/ using document root <none> and password file <none>
I20260926 21:41:15.750161 17301 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:15.750337 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:15.756894 17301 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 21:41:15.760068 17316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:15.761574 17301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 21:41:15.761775 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "b4266a1190644edbb96dabe8d7869715"
format_stamp: "Formatted at 2026-09-26 21:40:29 on dist-test-slave-lhhq"
server_key: "5b2c49cd5d03a713729e4be6437e7e6e"
server_key_iv: "0157526aa0e5d501b152ac78af96ae1c"
server_key_version: "encryptionkey@0"
I20260926 21:41:15.762759 17301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:15.780900 17301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:15.785357 17301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:15.785631 17301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:15.786638 17301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:15.787985 17301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:15.788080 17301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:15.788241 17301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:15.788326 17301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:15.821542 17301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39485
I20260926 21:41:15.821611 17428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39485 every 8 connection(s)
I20260926 21:41:15.823122 17301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:41:15.827960 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:15.826183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:43387)
0926 21:41:15.826905 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:15.827105 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":288,"threads_started":1}
I20260926 21:41:15.828553 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 17301
W20260926 21:41:15.828655 17429 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42309 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42309: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
I20260926 21:41:15.829612 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:15.829162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:48765)
0926 21:41:15.829368 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:15.829456 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 21:41:15.831060 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:15.830677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:57221)
0926 21:41:15.830830 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:15.830940 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 21:41:15.831319 17429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 21:41:15.925616 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:15.924745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:40361)
0926 21:41:15.925286 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:15.925392 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260926 21:41:16.375844 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:16.374923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:33573)
0926 21:41:16.375471 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:16.375575 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":237,"threads_started":1}
I20260926 21:41:16.832803 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:16.831866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:36705)
0926 21:41:16.832444 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:16.832545 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":300,"threads_started":1}
I20260926 21:41:16.927454 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:16.926674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:54751)
0926 21:41:16.927178 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:16.927264 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":249,"threads_started":1}
I20260926 21:41:17.377826 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:17.376931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:35253)
0926 21:41:17.377564 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:17.377656 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":398,"threads_started":1}
I20260926 21:41:17.834496 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:17.833708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:55435)
0926 21:41:17.834217 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:17.834314 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":280,"threads_started":1}
I20260926 21:41:17.929401 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:17.928517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:37095)
0926 21:41:17.929047 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:17.929133 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":346,"threads_started":1}
I20260926 21:41:18.379420 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:18.378747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:57813)
0926 21:41:18.379181 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:18.379255 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":212,"threads_started":1}
I20260926 21:41:18.836184 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:18.835394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:57077)
0926 21:41:18.835902 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:18.835988 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":260,"threads_started":1}
I20260926 21:41:18.931110 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:18.930304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:36575)
0926 21:41:18.930832 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:18.930922 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":269,"threads_started":1}
I20260926 21:41:19.381263 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:19.380455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:43509)
0926 21:41:19.380914 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:19.380996 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
I20260926 21:41:19.838109 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:19.837155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:34569)
0926 21:41:19.837764 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:19.837882 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
I20260926 21:41:19.932839 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:19.932124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:51273)
0926 21:41:19.932586 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:19.932667 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260926 21:41:20.382905 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:20.382173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:57027)
0926 21:41:20.382644 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:20.382725 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":245,"threads_started":1}
I20260926 21:41:20.839829 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:20.839099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:43215)
0926 21:41:20.839570 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:20.839653 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":229,"threads_started":1}
I20260926 21:41:20.934764 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:20.933967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:60595)
0926 21:41:20.934511 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:20.934599 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":311,"threads_started":1}
I20260926 21:41:21.384701 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:21.383887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:52251)
0926 21:41:21.384385 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:21.384524 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":267,"threads_started":1}
I20260926 21:41:21.841910 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:21.840984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:35735)
0926 21:41:21.841553 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:21.841662 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":338,"threads_started":1}
I20260926 21:41:21.936514 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:21.935675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:41463)
0926 21:41:21.936208 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:21.936302 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":238,"threads_started":1}
I20260926 21:41:22.386538 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:22.385730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:46899)
0926 21:41:22.386210 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:22.386304 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":212,"threads_started":1}
I20260926 21:41:22.843652 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:22.842813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:56279)
0926 21:41:22.843331 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:22.843427 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":242,"threads_started":1}
I20260926 21:41:22.938210 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:22.937496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:60045)
0926 21:41:22.937956 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:22.938047 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":225,"threads_started":1}
I20260926 21:41:23.388273 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:23.387553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:49853)
0926 21:41:23.388027 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:23.388110 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":238,"threads_started":1}
I20260926 21:41:23.845278 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:23.844582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:52493)
0926 21:41:23.845011 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:23.845089 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260926 21:41:23.939937 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:23.939104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:58023)
0926 21:41:23.939676 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:23.939760 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":300,"threads_started":1}
I20260926 21:41:24.389951 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:24.389129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:34803)
0926 21:41:24.389689 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:24.389768 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":334,"threads_started":1}
I20260926 21:41:24.847200 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:24.846259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:54429)
0926 21:41:24.846883 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:24.846992 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":270,"threads_started":1}
I20260926 21:41:24.941787 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:24.941003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:39659)
0926 21:41:24.941534 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:24.941627 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":303,"threads_started":1}
I20260926 21:41:25.391660 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:25.390908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:37091)
0926 21:41:25.391405 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:25.391489 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":275,"threads_started":1}
I20260926 21:41:25.849069 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:25.848276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:45397)
0926 21:41:25.848796 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:25.848897 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
I20260926 21:41:25.943547 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:25.942746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:45729)
0926 21:41:25.943233 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:25.943354 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260926 21:41:26.393602 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:26.392650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:35251)
0926 21:41:26.393264 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:26.393386 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":301,"threads_started":1}
I20260926 21:41:26.850871 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:26.850136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:43365)
0926 21:41:26.850591 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:26.850685 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":220,"threads_started":1}
I20260926 21:41:26.945518 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:26.944615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:35845)
0926 21:41:26.945133 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:26.945277 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":271,"threads_started":1}
I20260926 21:41:27.395598 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:27.394738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:44625)
0926 21:41:27.395217 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:27.395349 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":240,"threads_started":1}
I20260926 21:41:27.852708 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:27.851865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:58229)
0926 21:41:27.852434 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:27.852547 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":288,"threads_started":1}
I20260926 21:41:27.947269 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:27.946541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:59429)
0926 21:41:27.947000 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:27.947083 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":225,"threads_started":1}
I20260926 21:41:28.397445 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:28.396591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:36087)
0926 21:41:28.397097 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:28.397235 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":250,"threads_started":1}
I20260926 21:41:28.854638 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:28.853713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:39391)
0926 21:41:28.854259 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:28.854389 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":264,"threads_started":1}
I20260926 21:41:28.949003 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:28.948214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:53489)
0926 21:41:28.948714 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:28.948804 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
I20260926 21:41:29.399168 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:29.398403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:54809)
0926 21:41:29.398875 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:29.398986 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":253,"threads_started":1}
I20260926 21:41:29.856315 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:29.855534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:46803)
0926 21:41:29.855947 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:29.856030 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":211,"threads_started":1}
I20260926 21:41:29.950898 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:29.950187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.1:33153)
0926 21:41:29.950622 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:29.950711 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":222,"threads_started":1}
I20260926 21:41:30.401121 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:30.400432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.2:42955)
0926 21:41:30.400849 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:30.400941 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":209,"threads_started":1}
I20260926 21:41:30.839197 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 16999
I20260926 21:41:30.857317 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:30.840424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.0.97.1:39055)
0926 21:41:30.841611 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:30.841632 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:30.841684 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:41:30.843341 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:30.843350 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:30.843780 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:30.844084 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:30.845465 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:30.846745 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:30.847964 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:30.848638 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:30.848770 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:30.851307 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:30.851314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:30.851330 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:30.851377 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:30.854167 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:30.854821 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:30.854828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:30.854835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:30.855023 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:30.855334 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:30.855337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:30.855339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:30.855839 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:30.856101 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:30.856434 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 21:41:30.856845 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":734,"threads_started":1}
I20260926 21:41:30.858577 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:30.857500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42309 (local address 127.0.97.3:41557)
0926 21:41:30.858312 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:30.858410 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":632,"threads_started":1}
I20260926 21:41:30.859004 17162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:30.942615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 16999
I20260926 21:41:30.959950 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 17170
I20260926 21:41:30.978229 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:30.961098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.0.97.2:39097)
0926 21:41:30.961735 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:30.961755 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:30.961807 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:41:30.963398 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:30.963407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:30.963846 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:30.964150 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:30.965539 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:30.966989 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:30.968472 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:30.969244 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:30.969427 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:30.971974 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:30.971981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:30.971998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:30.972038 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:30.975112 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:30.975853 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:30.975859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:30.975866 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:30.976039 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:30.976364 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:30.976367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:30.976369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:30.976879 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:30.977102 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:30.977514 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 21:41:30.977943 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":270,"threads_started":1}
I20260926 21:41:30.979614 17293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:31.063277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 17170
I20260926 21:41:31.081027 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 17301
I20260926 21:41:31.098459 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:31.082187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.97.3:39485)
0926 21:41:31.082832 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:31.082846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:31.082888 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:41:31.084492 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:31.084501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:31.084945 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:31.085369 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:31.086692 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:31.087907 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:31.089133 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:31.089858 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:31.089993 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:31.092552 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:31.092559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:31.092574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:31.092616 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:31.095525 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:31.096226 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:31.096232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:31.096239 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:31.096394 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:31.096704 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:31.096707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:31.096709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:31.097266 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:31.097513 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:31.097836 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 21:41:31.098146 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":331,"threads_started":1}
I20260926 21:41:31.100185 17424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:31.182180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 17301
2026-09-26T21:41:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64607 ms)
[----------] 30 tests from SecurityITest (297875 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:41:31 dist-test-slave-lhhq krb5kdc[17486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:41:31 dist-test-slave-lhhq krb5kdc[17486](info): set up 2 sockets
Sep 26 21:41:31 dist-test-slave-lhhq krb5kdc[17486](info): commencing operation
krb5kdc: starting...
W20260926 21:41:32.289219 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:41:32 dist-test-slave-lhhq krb5kdc[17486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:41:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:41:32.549855 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41237
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:32.888019 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:32.888311 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:32.888480 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:32.895654 17499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:32.895762 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:32.895859 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:32.895967 17499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:32.896049 17499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:32.906739 17499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:32.908113 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:32.909471 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:32.921695 17504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:32.921726 17505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:32.921659 17507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:32.922237 17499 server_base.cc:1034] running on GCE node
I20260926 21:41:32.923177 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:32.924685 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:32.925920 17499 hybrid_clock.cc:648] HybridClock initialized: now 1790458892925861 us; error 54 us; skew 500 ppm
I20260926 21:41:32.926281 17499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:32 dist-test-slave-lhhq krb5kdc[17486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:32.929559 17499 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:41:32.931098 17499 webserver.cc:460] Webserver started at http://127.0.97.62:43379/ using document root <none> and password file <none>
I20260926 21:41:32.931841 17499 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:32.931998 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:32.932323 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:32.935504 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "32d34c70c3814d7c8b7195172957b0e0"
format_stamp: "Formatted at 2026-09-26 21:41:32 on dist-test-slave-lhhq"
server_key: "e38f374f7fbbcae423eeb0f85c311109"
server_key_iv: "991e8ee50d12050b692717d10de1c1e5"
server_key_version: "encryptionkey@0"
I20260926 21:41:32.936327 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "32d34c70c3814d7c8b7195172957b0e0"
format_stamp: "Formatted at 2026-09-26 21:41:32 on dist-test-slave-lhhq"
server_key: "e38f374f7fbbcae423eeb0f85c311109"
server_key_iv: "991e8ee50d12050b692717d10de1c1e5"
server_key_version: "encryptionkey@0"
I20260926 21:41:32.941901 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:41:32.945472 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:32.946895 17499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 21:41:32.947093 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "32d34c70c3814d7c8b7195172957b0e0"
format_stamp: "Formatted at 2026-09-26 21:41:32 on dist-test-slave-lhhq"
server_key: "e38f374f7fbbcae423eeb0f85c311109"
server_key_iv: "991e8ee50d12050b692717d10de1c1e5"
server_key_version: "encryptionkey@0"
I20260926 21:41:32.947301 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:32.968502 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:32.972643 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:32.972946 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:32.992084 17499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41237
I20260926 21:41:32.992116 17565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41237 every 8 connection(s)
I20260926 21:41:32.993613 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:32.994637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 17499
I20260926 21:41:32.994864 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:41:32.995522 388 external_mini_cluster.cc:1428] Setting key c9a51d655591e0ce09c49ad2761b3b23
I20260926 21:41:32.998232 17566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:41:33 dist-test-slave-lhhq krb5kdc[17486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:33.009284 17566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0: Bootstrap starting.
I20260926 21:41:33.012814 17566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:33.013998 17566 log.cc:824] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:33.017621 17566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0: No bootstrap required, opened a new log
I20260926 21:41:33.018718 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:32.997535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.97.62:41237)
0926 21:41:32.998356 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:32.998378 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:32.998421 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:41:33.000081 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:33.000101 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:33.000488 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:33.000897 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:33.002405 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:33.003756 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:33.005080 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:33.005822 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:33.006048 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:33.010487 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:33.010496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:33.010513 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:33.010560 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:33.013815 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:33.014748 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:33.014757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:33.014770 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:33.014954 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:33.015438 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:33.015444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:33.015449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:33.016169 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:33.016474 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:33.016856 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 21:41:33.017336 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":347,"threads_started":1}
I20260926 21:41:33.026247 17566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d34c70c3814d7c8b7195172957b0e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260926 21:41:33.026613 17566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:33.026686 17566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d34c70c3814d7c8b7195172957b0e0, State: Initialized, Role: FOLLOWER
I20260926 21:41:33.027179 17566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d34c70c3814d7c8b7195172957b0e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260926 21:41:33.027385 17566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:33.027483 17566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:33.027608 17566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:33.030048 17566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d34c70c3814d7c8b7195172957b0e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260926 21:41:33.030514 17566 leader_election.cc:302] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32d34c70c3814d7c8b7195172957b0e0; no voters:
I20260926 21:41:33.031051 17566 leader_election.cc:288] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:41:33.031214 17571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:33.031915 17571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [term 1 LEADER]: Becoming Leader. State: Replica: 32d34c70c3814d7c8b7195172957b0e0, State: Running, Role: LEADER
I20260926 21:41:33.032505 17571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d34c70c3814d7c8b7195172957b0e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260926 21:41:33.033351 17566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:33.035692 17572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32d34c70c3814d7c8b7195172957b0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d34c70c3814d7c8b7195172957b0e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } } }
I20260926 21:41:33.035816 17573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d34c70c3814d7c8b7195172957b0e0. Latest consensus state: current_term: 1 leader_uuid: "32d34c70c3814d7c8b7195172957b0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d34c70c3814d7c8b7195172957b0e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } } }
I20260926 21:41:33.036331 17573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:33.036433 17572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:33.037964 17577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:33.043268 17577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:33.053308 17577 catalog_manager.cc:1347] Generated new cluster ID: 14891d37e7494b7c9f8361939aae93d8
I20260926 21:41:33.053431 17577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:33.070400 17577 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:41:33.072381 17577 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:33.087927 17577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0: Generated new TSK 0
I20260926 21:41:33.088865 17577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:41:33.146729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:33.487182 17592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:33.487474 17592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:33.487637 17592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:33.495005 17592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:33.495132 17592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:33.495325 17592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:41:33.505386 17592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:33.506713 17592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:33.508072 17592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:33.521747 17598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:33.521730 17597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:33.522043 17600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:33.523114 17592 server_base.cc:1034] running on GCE node
I20260926 21:41:33.523828 17592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:33.524827 17592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:33.526022 17592 hybrid_clock.cc:648] HybridClock initialized: now 1790458893525952 us; error 62 us; skew 500 ppm
I20260926 21:41:33.526347 17592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:33 dist-test-slave-lhhq krb5kdc[17486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:33.529606 17592 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:41:33.531124 17592 webserver.cc:460] Webserver started at http://127.0.97.1:37245/ using document root <none> and password file <none>
I20260926 21:41:33.531874 17592 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:33.532017 17592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:33.532320 17592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:33.535455 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "964ea21fde2a4941b0304ae73b97077a"
format_stamp: "Formatted at 2026-09-26 21:41:33 on dist-test-slave-lhhq"
server_key: "a1afe6489a9394061baa1ee4d9ccce28"
server_key_iv: "d6615d0e833eb92ce5ceea95b54e5aaa"
server_key_version: "encryptionkey@0"
I20260926 21:41:33.536268 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "964ea21fde2a4941b0304ae73b97077a"
format_stamp: "Formatted at 2026-09-26 21:41:33 on dist-test-slave-lhhq"
server_key: "a1afe6489a9394061baa1ee4d9ccce28"
server_key_iv: "d6615d0e833eb92ce5ceea95b54e5aaa"
server_key_version: "encryptionkey@0"
I20260926 21:41:33.541775 17592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:41:33.545383 17607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:33.546748 17592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:41:33.546922 17592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "964ea21fde2a4941b0304ae73b97077a"
format_stamp: "Formatted at 2026-09-26 21:41:33 on dist-test-slave-lhhq"
server_key: "a1afe6489a9394061baa1ee4d9ccce28"
server_key_iv: "d6615d0e833eb92ce5ceea95b54e5aaa"
server_key_version: "encryptionkey@0"
I20260926 21:41:33.547127 17592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:33.562332 17592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:33.566553 17592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:33.566843 17592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:33.567808 17592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:33.569094 17592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:33.569197 17592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:33.569375 17592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:33.569456 17592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:33.603215 17592 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41449
I20260926 21:41:33.603267 17719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41449 every 8 connection(s)
I20260926 21:41:33.604538 17592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:41:33.611431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 17592
I20260926 21:41:33.611665 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:41:33.612291 388 external_mini_cluster.cc:1428] Setting key 8b85cc62b0b9be2c318034cef3e6e402
Sep 26 21:41:33 dist-test-slave-lhhq krb5kdc[17486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:33.626672 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:33.607240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58661 (local address 127.0.97.62:41237)
0926 21:41:33.607684 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:33.607689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:33.608711 (+ 1022us) server_negotiation.cc:407] Connection header received
0926 21:41:33.610645 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:33.610651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:33.610748 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:33.610872 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:33.612576 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:33.613397 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:33.615309 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:33.616073 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:33.616227 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:33.619861 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:33.619871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:33.619879 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:33.619929 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:33.623072 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:33.624149 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:33.624153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:33.624156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:33.624294 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:33.624813 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:33.624816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:33.624818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:33.625085 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:33.625272 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:33.626221 (+ 949us) server_negotiation.cc:299] Negotiation successful
0926 21:41:33.626410 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":236,"threads_started":1}
I20260926 21:41:33.628283 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:33.607843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41237 (local address 127.0.97.1:58661)
0926 21:41:33.608540 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:33.608585 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:33.610361 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:33.611130 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:33.611138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:33.611747 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:33.612365 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:33.612410 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:33.613603 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:33.613608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:33.615145 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:33.615158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:33.616307 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:33.616312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:33.616535 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:33.617315 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:41:33.617338 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:33.619600 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:33.623270 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:33.623282 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:33.623304 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:33.624005 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:33.624416 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:33.624424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:33.624430 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:33.624704 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:33.625244 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:33.625256 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:33.625796 (+ 540us) client_negotiation.cc:769] Sending connection context
0926 21:41:33.626129 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 21:41:33.626762 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":274,"threads_started":1}
I20260926 21:41:33.630046 17720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41237
I20260926 21:41:33.630445 17720 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:33.631233 17720 heartbeater.cc:511] Master 127.0.97.62:41237 requested a full tablet report, sending...
I20260926 21:41:33.633615 17531 ts_manager.cc:194] Registered new tserver with Master: 964ea21fde2a4941b0304ae73b97077a (127.0.97.1:41449)
I20260926 21:41:33.635169 17531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58661
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:41:33.671219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:34.002228 17726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:34.002519 17726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:34.002682 17726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:34.010030 17726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:34.010154 17726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:34.010362 17726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:41:34.020274 17726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:34.021727 17726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:34.023110 17726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:34.036244 17731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:34.036435 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:34.036715 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:34.037156 17726 server_base.cc:1034] running on GCE node
I20260926 21:41:34.037777 17726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:34.038763 17726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:34.040000 17726 hybrid_clock.cc:648] HybridClock initialized: now 1790458894039938 us; error 54 us; skew 500 ppm
I20260926 21:41:34.040382 17726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:34 dist-test-slave-lhhq krb5kdc[17486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:34.043834 17726 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:41:34.045313 17726 webserver.cc:460] Webserver started at http://127.0.97.2:36123/ using document root <none> and password file <none>
I20260926 21:41:34.045992 17726 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:34.046136 17726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:34.046443 17726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:34.049474 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "bf620e86c7584b6eb662dc4535defd8e"
format_stamp: "Formatted at 2026-09-26 21:41:34 on dist-test-slave-lhhq"
server_key: "6f8d3fa6282d6826420715197018ab55"
server_key_iv: "0da8b6063ea8b4a8277e7e54f9cab81b"
server_key_version: "encryptionkey@0"
I20260926 21:41:34.050400 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bf620e86c7584b6eb662dc4535defd8e"
format_stamp: "Formatted at 2026-09-26 21:41:34 on dist-test-slave-lhhq"
server_key: "6f8d3fa6282d6826420715197018ab55"
server_key_iv: "0da8b6063ea8b4a8277e7e54f9cab81b"
server_key_version: "encryptionkey@0"
I20260926 21:41:34.055771 17726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 21:41:34.059162 17741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:34.060523 17726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:41:34.060694 17726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "bf620e86c7584b6eb662dc4535defd8e"
format_stamp: "Formatted at 2026-09-26 21:41:34 on dist-test-slave-lhhq"
server_key: "6f8d3fa6282d6826420715197018ab55"
server_key_iv: "0da8b6063ea8b4a8277e7e54f9cab81b"
server_key_version: "encryptionkey@0"
I20260926 21:41:34.060904 17726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:34.076561 17726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:34.080971 17726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:34.081319 17726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:34.082460 17726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:34.083925 17726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:34.084035 17726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:34.084208 17726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:34.084291 17726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:34.119309 17726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44033
I20260926 21:41:34.119403 17853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44033 every 8 connection(s)
I20260926 21:41:34.120628 17726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:41:34.125435 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 17726
I20260926 21:41:34.125625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:41:34.126175 388 external_mini_cluster.cc:1428] Setting key 45a7158c0207420c682d3f335a32817f
Sep 26 21:41:34 dist-test-slave-lhhq krb5kdc[17486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:34.143397 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.123374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39709 (local address 127.0.97.62:41237)
0926 21:41:34.123587 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:34.123593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:34.124793 (+ 1200us) server_negotiation.cc:407] Connection header received
0926 21:41:34.127286 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:34.127291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:34.127377 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:34.127486 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:34.129430 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.130158 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.132629 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.133060 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.133253 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.137488 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:34.137493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:34.137497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:34.137530 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:34.140289 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:34.141007 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:34.141017 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:34.141019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:34.141143 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:34.141674 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:34.141679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:34.141682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:34.141949 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:34.142103 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:34.142801 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 21:41:34.143035 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":76}
I20260926 21:41:34.144389 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.123938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41237 (local address 127.0.97.2:39709)
0926 21:41:34.124630 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:34.124667 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:34.126981 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:34.127679 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:34.127687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:34.128359 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:34.129235 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.129264 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.130316 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.130323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.132466 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.132482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.133323 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.133331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.133692 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.134502 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:41:34.134538 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:34.137258 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:34.140447 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:34.140456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:34.140473 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:34.140876 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:34.141325 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:34.141330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:34.141334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:34.141545 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:34.142076 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:34.142083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:34.142431 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 21:41:34.142697 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 21:41:34.143225 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":272,"threads_started":1}
I20260926 21:41:34.145967 17854 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41237
I20260926 21:41:34.146278 17854 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:34.147014 17854 heartbeater.cc:511] Master 127.0.97.62:41237 requested a full tablet report, sending...
I20260926 21:41:34.148519 17531 ts_manager.cc:194] Registered new tserver with Master: bf620e86c7584b6eb662dc4535defd8e (127.0.97.2:44033)
I20260926 21:41:34.149415 17531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39709
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:41:34.181532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:34.507588 17859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:34.507900 17859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:34.507997 17859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:34.515419 17859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:34.515547 17859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:34.515666 17859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:41:34.525895 17859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:34.527179 17859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:34.528441 17859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:34.542013 17864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:34.542040 17865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:34.542083 17867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:34.542800 17859 server_base.cc:1034] running on GCE node
I20260926 21:41:34.543403 17859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:34.544405 17859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:34.545651 17859 hybrid_clock.cc:648] HybridClock initialized: now 1790458894545582 us; error 56 us; skew 500 ppm
I20260926 21:41:34.545995 17859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:34 dist-test-slave-lhhq krb5kdc[17486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:34.553644 17859 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:41:34.555441 17859 webserver.cc:460] Webserver started at http://127.0.97.3:45221/ using document root <none> and password file <none>
I20260926 21:41:34.556185 17859 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:34.556335 17859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:34.556667 17859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:34.559939 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "9af2a2b4165f4f54b8f6f6051c44b154"
format_stamp: "Formatted at 2026-09-26 21:41:34 on dist-test-slave-lhhq"
server_key: "79e0765eac4d2f9c63075e349385782e"
server_key_iv: "1dd93f604e760f338ce82289f707b2a2"
server_key_version: "encryptionkey@0"
I20260926 21:41:34.560751 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9af2a2b4165f4f54b8f6f6051c44b154"
format_stamp: "Formatted at 2026-09-26 21:41:34 on dist-test-slave-lhhq"
server_key: "79e0765eac4d2f9c63075e349385782e"
server_key_iv: "1dd93f604e760f338ce82289f707b2a2"
server_key_version: "encryptionkey@0"
I20260926 21:41:34.566236 17859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:41:34.569986 17874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:34.571597 17859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:41:34.571777 17859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "9af2a2b4165f4f54b8f6f6051c44b154"
format_stamp: "Formatted at 2026-09-26 21:41:34 on dist-test-slave-lhhq"
server_key: "79e0765eac4d2f9c63075e349385782e"
server_key_iv: "1dd93f604e760f338ce82289f707b2a2"
server_key_version: "encryptionkey@0"
I20260926 21:41:34.571985 17859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:34.596074 17859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:34.600472 17859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:34.600781 17859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:34.601863 17859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:34.603295 17859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:34.603379 17859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:34.603510 17859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:34.603561 17859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:34.637324 17859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39101
I20260926 21:41:34.637429 17986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39101 every 8 connection(s)
I20260926 21:41:34.638236 17720 heartbeater.cc:503] Master 127.0.97.62:41237 was elected leader, sending a full tablet report...
I20260926 21:41:34.638816 17859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:41:34.646504 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 17859
I20260926 21:41:34.646706 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:41:34.647245 388 external_mini_cluster.cc:1428] Setting key 53ca5c74866705b6492d741eb9af5204
Sep 26 21:41:34 dist-test-slave-lhhq krb5kdc[17486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:34.660395 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.641703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58875 (local address 127.0.97.62:41237)
0926 21:41:34.641880 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:34.641886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:34.643147 (+ 1261us) server_negotiation.cc:407] Connection header received
0926 21:41:34.645024 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:34.645028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:34.645097 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:34.645232 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:34.647092 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.648037 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.650143 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.650715 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.650884 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.654497 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:34.654502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:34.654505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:34.654542 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:34.657486 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:34.658180 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:34.658184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:34.658187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:34.658347 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:34.658740 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:34.658743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:34.658747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:34.659050 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:34.659178 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:34.659924 (+ 746us) server_negotiation.cc:299] Negotiation successful
0926 21:41:34.660104 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:41:34.661854 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.642289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41237 (local address 127.0.97.3:58875)
0926 21:41:34.642985 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:34.643023 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:34.644694 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:34.645465 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:34.645472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:34.646003 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:34.646878 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.646904 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.648199 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.648204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.649975 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.650008 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.650971 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.650997 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.651242 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.652056 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:41:34.652078 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:34.654270 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:34.657604 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:34.657611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:34.657625 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:34.658044 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:34.658448 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:34.658451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:34.658454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:34.658623 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:34.659206 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:34.659212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:34.659544 (+ 332us) client_negotiation.cc:769] Sending connection context
0926 21:41:34.659781 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 21:41:34.660179 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":274,"threads_started":1}
I20260926 21:41:34.663339 17987 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41237
I20260926 21:41:34.663652 17987 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:34.664355 17987 heartbeater.cc:511] Master 127.0.97.62:41237 requested a full tablet report, sending...
I20260926 21:41:34.666006 17531 ts_manager.cc:194] Registered new tserver with Master: 9af2a2b4165f4f54b8f6f6051c44b154 (127.0.97.3:39101)
I20260926 21:41:34.666919 17531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58875
I20260926 21:41:34.675396 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:41:34.692517 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.680083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.0.97.62:41237)
0926 21:41:34.680258 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:34.680262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:34.680746 (+ 484us) server_negotiation.cc:407] Connection header received
0926 21:41:34.680909 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:34.680912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:34.680976 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:34.681062 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:34.682506 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.683199 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.684583 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.685007 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.685114 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.687483 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:34.687488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:34.687491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:34.687525 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:34.690252 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:34.690885 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:34.690889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:34.690892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:34.691022 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:34.691346 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:34.691349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:34.691351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:34.691658 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:34.691842 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:34.692123 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 21:41:34.692290 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:41:34.696652 17531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:41:34.703426 17531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:41:34.736213 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.716027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.0.97.1:41449)
0926 21:41:34.727203 (+ 11176us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:34.727212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:34.727241 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:41:34.727357 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:34.727368 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:34.727680 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:34.727927 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:34.728699 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.730371 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.733318 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.734905 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.735034 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.735294 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:34.735687 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 21:41:34.735874 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10984,"thread_start_us":11028,"threads_started":1}
I20260926 21:41:34.736244 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.715742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41449 (local address 127.0.0.1:34296)
0926 21:41:34.716466 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:34.716513 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:34.716678 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:34.727959 (+ 11281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:34.727968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:34.728006 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:34.728456 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.728476 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.730519 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.730524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.733117 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.733136 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.735129 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.735135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.735342 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.735399 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 21:41:34.735552 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 21:41:34.735769 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":60,"thread_start_us":316,"threads_started":1}
I20260926 21:41:34.736691 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.716575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39101 (local address 127.0.0.1:52572)
0926 21:41:34.726767 (+ 10192us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:34.726796 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:34.726982 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:34.728894 (+ 1912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:34.728899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:34.728926 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:34.729425 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.729439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.731354 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.731370 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.733581 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.733594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.735814 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.735824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.736004 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.736037 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:41:34.736114 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 21:41:34.736268 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10037,"mutex_wait_us":65,"thread_start_us":9878,"threads_started":1}
I20260926 21:41:34.737172 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.717292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.0.97.3:39101)
0926 21:41:34.728211 (+ 10919us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:34.728224 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:34.728267 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:41:34.728374 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:34.728381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:34.728641 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:34.729630 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:34.730056 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.731212 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.733776 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.735201 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.735343 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.736462 (+ 1119us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:34.736577 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 21:41:34.736797 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10713,"thread_start_us":10543,"threads_started":1}
I20260926 21:41:34.737841 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.716895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44033 (local address 127.0.0.1:52550)
0926 21:41:34.728970 (+ 12075us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:34.728992 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:34.729121 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:34.729806 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:34.729812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:34.729835 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:34.730167 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.730175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.732104 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.732110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.734069 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.734088 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.736250 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.736257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.737287 (+ 1030us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.737323 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 21:41:34.737420 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 21:41:34.737544 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11935,"thread_start_us":11437,"threads_started":1}
I20260926 21:41:34.738432 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.717253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.0.97.2:44033)
0926 21:41:34.727649 (+ 10396us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:34.727659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:34.729276 (+ 1617us) server_negotiation.cc:407] Connection header received
0926 21:41:34.729383 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:34.729392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:34.729614 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:34.729820 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:34.730332 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.731917 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.734245 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.735754 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.735871 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.736078 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:34.737820 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0926 21:41:34.738014 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10252,"thread_start_us":10157,"threads_started":1}
I20260926 21:41:34.741446 17922 tablet_service.cc:1467] Processing CreateTablet for tablet 191aff7724f8479db99312071849872d (DEFAULT_TABLE table=test-table [id=a62379abafba48f0974796115477e9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:34.741456 17655 tablet_service.cc:1467] Processing CreateTablet for tablet 191aff7724f8479db99312071849872d (DEFAULT_TABLE table=test-table [id=a62379abafba48f0974796115477e9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:34.741991 17789 tablet_service.cc:1467] Processing CreateTablet for tablet 191aff7724f8479db99312071849872d (DEFAULT_TABLE table=test-table [id=a62379abafba48f0974796115477e9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:34.743000 17655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191aff7724f8479db99312071849872d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:34.743007 17922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191aff7724f8479db99312071849872d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:34.743184 17789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191aff7724f8479db99312071849872d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:34.755234 18005 tablet_bootstrap.cc:492] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: Bootstrap starting.
I20260926 21:41:34.757856 18007 tablet_bootstrap.cc:492] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: Bootstrap starting.
I20260926 21:41:34.759135 18005 tablet_bootstrap.cc:654] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:34.760552 18005 log.cc:824] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:34.761411 18007 tablet_bootstrap.cc:654] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:34.761663 18006 tablet_bootstrap.cc:492] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: Bootstrap starting.
I20260926 21:41:34.762661 18007 log.cc:824] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:34.764154 18005 tablet_bootstrap.cc:492] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: No bootstrap required, opened a new log
I20260926 21:41:34.764463 18005 ts_tablet_manager.cc:1397] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260926 21:41:34.764974 18006 tablet_bootstrap.cc:654] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:34.766018 18006 log.cc:824] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:34.766131 18007 tablet_bootstrap.cc:492] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: No bootstrap required, opened a new log
I20260926 21:41:34.766410 18007 ts_tablet_manager.cc:1397] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260926 21:41:34.769142 18006 tablet_bootstrap.cc:492] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: No bootstrap required, opened a new log
I20260926 21:41:34.769435 18006 ts_tablet_manager.cc:1397] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260926 21:41:34.775226 18005 raft_consensus.cc:348] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } }
I20260926 21:41:34.775794 18005 raft_consensus.cc:374] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:34.775902 18005 raft_consensus.cc:729] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf620e86c7584b6eb662dc4535defd8e, State: Initialized, Role: FOLLOWER
I20260926 21:41:34.776549 18005 consensus_queue.cc:260] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } }
I20260926 21:41:34.776929 18006 raft_consensus.cc:348] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } }
I20260926 21:41:34.776976 18007 raft_consensus.cc:348] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } }
I20260926 21:41:34.777395 18006 raft_consensus.cc:374] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:34.777499 18006 raft_consensus.cc:729] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964ea21fde2a4941b0304ae73b97077a, State: Initialized, Role: FOLLOWER
I20260926 21:41:34.777518 18007 raft_consensus.cc:374] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:34.777611 18007 raft_consensus.cc:729] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af2a2b4165f4f54b8f6f6051c44b154, State: Initialized, Role: FOLLOWER
I20260926 21:41:34.778102 17854 heartbeater.cc:503] Master 127.0.97.62:41237 was elected leader, sending a full tablet report...
I20260926 21:41:34.778234 18007 consensus_queue.cc:260] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } }
I20260926 21:41:34.778513 18005 ts_tablet_manager.cc:1428] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: Time spent starting tablet: real 0.014s user 0.013s sys 0.003s
I20260926 21:41:34.778970 18006 consensus_queue.cc:260] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } }
I20260926 21:41:34.781859 18007 ts_tablet_manager.cc:1428] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 21:41:34.783468 17987 heartbeater.cc:503] Master 127.0.97.62:41237 was elected leader, sending a full tablet report...
I20260926 21:41:34.784703 18006 ts_tablet_manager.cc:1428] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
W20260926 21:41:34.856976 17721 tablet.cc:2369] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:41:34.872751 17855 tablet.cc:2369] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:41:34.890650 17988 tablet.cc:2369] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:34.932508 18011 raft_consensus.cc:482] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:41:34.932829 18011 raft_consensus.cc:504] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } }
I20260926 21:41:34.934854 18011 leader_election.cc:288] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 964ea21fde2a4941b0304ae73b97077a (127.0.97.1:41449), 9af2a2b4165f4f54b8f6f6051c44b154 (127.0.97.3:39101)
I20260926 21:41:34.941977 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.935297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51287 (local address 127.0.97.1:41449)
0926 21:41:34.935547 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:34.935557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:34.936228 (+ 671us) server_negotiation.cc:407] Connection header received
0926 21:41:34.936424 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:34.936428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:34.936498 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:34.936593 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:34.937142 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.938099 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.940216 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.940986 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.941142 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.941275 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:34.941602 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 21:41:34.941727 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 21:41:34.942005 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.935277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52489 (local address 127.0.97.3:39101)
0926 21:41:34.935526 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:34.935535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:34.936312 (+ 777us) server_negotiation.cc:407] Connection header received
0926 21:41:34.936531 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:34.936535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:34.936592 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:34.936687 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:34.937378 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.938122 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.940246 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.941016 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.941166 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.941306 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:34.941628 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 21:41:34.941752 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:41:34.942150 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.935281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39101 (local address 127.0.97.2:52489)
0926 21:41:34.936217 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:34.936235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:34.936379 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:34.936703 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:34.936708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:34.936745 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:34.937173 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.937233 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.938254 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.938260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.940103 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.940115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.941263 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.941269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.941454 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.941477 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:41:34.941592 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 21:41:34.941714 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":113,"thread_start_us":562,"threads_started":1}
I20260926 21:41:34.942520 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.935161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41449 (local address 127.0.97.2:51287)
0926 21:41:34.936115 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:34.936135 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:34.936262 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:34.936609 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:34.936614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:34.936635 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:34.936987 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.936996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.938224 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.938232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.940064 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:34.940076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.941252 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.941260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:34.941426 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.941453 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 21:41:34.941577 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 21:41:34.941709 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":93,"thread_start_us":586,"threads_started":1}
I20260926 21:41:34.943192 17942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191aff7724f8479db99312071849872d" candidate_uuid: "bf620e86c7584b6eb662dc4535defd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" is_pre_election: true
I20260926 21:41:34.943383 17675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191aff7724f8479db99312071849872d" candidate_uuid: "bf620e86c7584b6eb662dc4535defd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964ea21fde2a4941b0304ae73b97077a" is_pre_election: true
I20260926 21:41:34.943706 17942 raft_consensus.cc:2454] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf620e86c7584b6eb662dc4535defd8e in term 0.
I20260926 21:41:34.943919 17675 raft_consensus.cc:2454] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf620e86c7584b6eb662dc4535defd8e in term 0.
I20260926 21:41:34.944382 17743 leader_election.cc:302] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9af2a2b4165f4f54b8f6f6051c44b154, bf620e86c7584b6eb662dc4535defd8e; no voters:
I20260926 21:41:34.944796 18011 raft_consensus.cc:2781] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:41:34.944950 18011 raft_consensus.cc:482] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:41:34.945053 18011 raft_consensus.cc:3037] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:34.947230 18011 raft_consensus.cc:504] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } }
I20260926 21:41:34.947844 18011 leader_election.cc:288] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [CANDIDATE]: Term 1 election: Requested vote from peers 964ea21fde2a4941b0304ae73b97077a (127.0.97.1:41449), 9af2a2b4165f4f54b8f6f6051c44b154 (127.0.97.3:39101)
I20260926 21:41:34.948253 17675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191aff7724f8479db99312071849872d" candidate_uuid: "bf620e86c7584b6eb662dc4535defd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964ea21fde2a4941b0304ae73b97077a"
I20260926 21:41:34.948297 17942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191aff7724f8479db99312071849872d" candidate_uuid: "bf620e86c7584b6eb662dc4535defd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9af2a2b4165f4f54b8f6f6051c44b154"
I20260926 21:41:34.948485 17942 raft_consensus.cc:3037] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:34.948489 17675 raft_consensus.cc:3037] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:34.951781 17942 raft_consensus.cc:2454] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf620e86c7584b6eb662dc4535defd8e in term 1.
I20260926 21:41:34.951810 17675 raft_consensus.cc:2454] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf620e86c7584b6eb662dc4535defd8e in term 1.
I20260926 21:41:34.952205 17744 leader_election.cc:302] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 964ea21fde2a4941b0304ae73b97077a, bf620e86c7584b6eb662dc4535defd8e; no voters:
I20260926 21:41:34.952541 18011 raft_consensus.cc:2781] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:34.952971 18011 raft_consensus.cc:686] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 1 LEADER]: Becoming Leader. State: Replica: bf620e86c7584b6eb662dc4535defd8e, State: Running, Role: LEADER
I20260926 21:41:34.953734 18011 consensus_queue.cc:237] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } }
I20260926 21:41:34.958669 17531 catalog_manager.cc:5495] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf620e86c7584b6eb662dc4535defd8e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "bf620e86c7584b6eb662dc4535defd8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf620e86c7584b6eb662dc4535defd8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 } health_report { overall_health: UNKNOWN } } }
I20260926 21:41:34.983174 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:34.977846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.0.97.2:44033)
0926 21:41:34.978107 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:34.978112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:34.978129 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:41:34.978246 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:34.978252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:34.978345 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:34.978478 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:41:34.978781 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.979648 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.981086 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:34.981552 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:34.981647 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:34.982103 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:41:34.982601 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:41:34.982716 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:34.982796 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 21:41:34.982858 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 21:41:34.991168 17942 raft_consensus.cc:1261] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [term 1 FOLLOWER]: Refusing update from remote peer bf620e86c7584b6eb662dc4535defd8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:34.991204 17675 raft_consensus.cc:1261] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [term 1 FOLLOWER]: Refusing update from remote peer bf620e86c7584b6eb662dc4535defd8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:34.992274 18016 consensus_queue.cc:1035] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9af2a2b4165f4f54b8f6f6051c44b154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:34.992640 18011 consensus_queue.cc:1035] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "964ea21fde2a4941b0304ae73b97077a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:35.025098 18021 mvcc.cc:205] Tried to move back new op lower bound from 7333719633874087936 to 7333719633734397952. Current Snapshot: MvccSnapshot[applied={T|T < 7333719633874087936 or (T in {7333719633874087936})}]
I20260926 21:41:35.031986 17529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:35.032298 17529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:35.036324 17529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32d34c70c3814d7c8b7195172957b0e0: Sending DeleteTablet for 3 replicas of tablet 191aff7724f8479db99312071849872d
I20260926 21:41:35.037535 17655 tablet_service.cc:1514] Processing DeleteTablet for tablet 191aff7724f8479db99312071849872d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34296
I20260926 21:41:35.037676 17922 tablet_service.cc:1514] Processing DeleteTablet for tablet 191aff7724f8479db99312071849872d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52572
I20260926 21:41:35.038343 17789 tablet_service.cc:1514] Processing DeleteTablet for tablet 191aff7724f8479db99312071849872d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52550
I20260926 21:41:35.039383 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 17592
I20260926 21:41:35.061836 18030 tablet_replica.cc:330] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: stopping tablet replica
Sep 26 21:41:35 dist-test-slave-lhhq krb5kdc[17486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:41:35.062670 18030 raft_consensus.cc:2229] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:41:35.063611 18030 raft_consensus.cc:2258] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:35.066598 18029 tablet_replica.cc:330] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: stopping tablet replica
I20260926 21:41:35.067051 18029 raft_consensus.cc:2229] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:35.067371 18030 ts_tablet_manager.cc:1905] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:35.067926 18028 tablet_replica.cc:330] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: stopping tablet replica
I20260926 21:41:35.068349 18028 raft_consensus.cc:2229] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:35.068895 18028 raft_consensus.cc:2258] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:35.069597 18029 raft_consensus.cc:2258] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:35.071939 18028 ts_tablet_manager.cc:1905] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:35.072310 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:35.040728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.0.97.1:41449)
0926 21:41:35.040900 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:35.040906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:35.051330 (+ 10424us) server_negotiation.cc:407] Connection header received
0926 21:41:35.051681 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:35.051688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:35.051797 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:35.051885 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:35.053788 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:35.054981 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:35.056918 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:35.057622 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:35.057721 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:35.063352 (+ 5631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:35.063371 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:35.063391 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:35.063445 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:35.067888 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:35.068905 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:35.068916 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:35.068921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:35.069097 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:35.069663 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:35.069671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:35.069674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:35.070100 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:35.070440 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:35.071640 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0926 21:41:35.071853 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:41:35.073137 17715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:35.073678 18029 ts_tablet_manager.cc:1905] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:35.079734 18030 ts_tablet_manager.cc:1918] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:35.079934 18030 log.cc:1196] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/191aff7724f8479db99312071849872d
I20260926 21:41:35.080500 18030 ts_tablet_manager.cc:1939] T 191aff7724f8479db99312071849872d P bf620e86c7584b6eb662dc4535defd8e: Deleting consensus metadata
I20260926 21:41:35.082553 17517 catalog_manager.cc:4841] TS bf620e86c7584b6eb662dc4535defd8e (127.0.97.2:44033): tablet 191aff7724f8479db99312071849872d (table test-table [id=a62379abafba48f0974796115477e9e8]) successfully deleted
I20260926 21:41:35.088421 18029 ts_tablet_manager.cc:1918] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:35.088608 18029 log.cc:1196] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/191aff7724f8479db99312071849872d
I20260926 21:41:35.089144 18029 ts_tablet_manager.cc:1939] T 191aff7724f8479db99312071849872d P 9af2a2b4165f4f54b8f6f6051c44b154: Deleting consensus metadata
I20260926 21:41:35.091204 17516 catalog_manager.cc:4841] TS 9af2a2b4165f4f54b8f6f6051c44b154 (127.0.97.3:39101): tablet 191aff7724f8479db99312071849872d (table test-table [id=a62379abafba48f0974796115477e9e8]) successfully deleted
I20260926 21:41:35.168912 18028 ts_tablet_manager.cc:1918] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:35.169101 18028 log.cc:1196] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/191aff7724f8479db99312071849872d
I20260926 21:41:35.169610 18028 ts_tablet_manager.cc:1939] T 191aff7724f8479db99312071849872d P 964ea21fde2a4941b0304ae73b97077a: Deleting consensus metadata
I20260926 21:41:35.171283 17517 catalog_manager.cc:4841] TS 964ea21fde2a4941b0304ae73b97077a (127.0.97.1:41449): tablet 191aff7724f8479db99312071849872d (table test-table [id=a62379abafba48f0974796115477e9e8]) successfully deleted
I20260926 21:41:35.171628 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 17592
I20260926 21:41:35.190328 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 17726
Sep 26 21:41:35 dist-test-slave-lhhq krb5kdc[17486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:41:35.208395 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:35.191213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.0.97.2:44033)
0926 21:41:35.191401 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:35.191408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:35.191477 (+ 69us) server_negotiation.cc:407] Connection header received
0926 21:41:35.191646 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:35.191652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:35.191737 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:35.191850 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:35.193472 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:35.194714 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:35.196074 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:35.196737 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:35.196863 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:35.201581 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:35.201600 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:35.201640 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:35.201711 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:35.205486 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:35.206396 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:35.206403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:35.206406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:35.206525 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:35.206949 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:35.206952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:35.206955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:35.207303 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:35.207552 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:35.207870 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 21:41:35.208039 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:41:35.209093 17849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:35.321627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 17726
I20260926 21:41:35.340734 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 17859
Sep 26 21:41:35 dist-test-slave-lhhq krb5kdc[17486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:41:35.355990 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:35.341517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.0.97.3:39101)
0926 21:41:35.341713 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:35.341717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:35.341734 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:41:35.341892 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:35.341896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:35.341967 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:35.342051 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:35.343297 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:35.344003 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:35.345166 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:35.345647 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:35.345754 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:35.350182 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:35.350196 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:35.350212 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:35.350280 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:35.353479 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:35.354135 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:35.354142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:35.354144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:35.354297 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:35.354605 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:35.354608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:35.354609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:35.354959 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:35.355173 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:35.355445 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 21:41:35.355611 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:41:35.356702 17982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:35.447350 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 17859
I20260926 21:41:35.464730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 17499
I20260926 21:41:35.465900 17561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:35.576249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 17499
2026-09-26T21:41:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4396 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T21:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:41:35Z Disabled control of system clock
I20260926 21:41:35.651229 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35215
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:41:35.993572 18039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:35.993866 18039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:35.994010 18039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:36.001391 18039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:36.001552 18039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:36.001624 18039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:36.001731 18039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:36.001814 18039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:36.010996 18039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:36.012341 18039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:36.014281 18039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:36.025998 18044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:36.025970 18045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:36.026055 18047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:36.027081 18039 server_base.cc:1034] running on GCE node
I20260926 21:41:36.027891 18039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:36.029253 18039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:36.030480 18039 hybrid_clock.cc:648] HybridClock initialized: now 1790458896030420 us; error 57 us; skew 500 ppm
I20260926 21:41:36.030784 18039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:36.033008 18039 webserver.cc:460] Webserver started at http://127.0.97.62:44401/ using document root <none> and password file <none>
I20260926 21:41:36.033726 18039 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:36.033813 18039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:36.034075 18039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:36.036873 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "22935ee938c04aa192568aca46eca509"
format_stamp: "Formatted at 2026-09-26 21:41:36 on dist-test-slave-lhhq"
server_key: "35e86fa77006718670c1fce36ebd5c80"
server_key_iv: "b7b972d843b6c119ca4715c025e9d0b3"
server_key_version: "encryptionkey@0"
I20260926 21:41:36.037694 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "22935ee938c04aa192568aca46eca509"
format_stamp: "Formatted at 2026-09-26 21:41:36 on dist-test-slave-lhhq"
server_key: "35e86fa77006718670c1fce36ebd5c80"
server_key_iv: "b7b972d843b6c119ca4715c025e9d0b3"
server_key_version: "encryptionkey@0"
I20260926 21:41:36.042739 18039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 21:41:36.046026 18053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:36.047348 18039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 21:41:36.047542 18039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "22935ee938c04aa192568aca46eca509"
format_stamp: "Formatted at 2026-09-26 21:41:36 on dist-test-slave-lhhq"
server_key: "35e86fa77006718670c1fce36ebd5c80"
server_key_iv: "b7b972d843b6c119ca4715c025e9d0b3"
server_key_version: "encryptionkey@0"
I20260926 21:41:36.047753 18039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:36.069561 18039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:36.070454 18039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:36.070760 18039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:36.089850 18039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35215
I20260926 21:41:36.089857 18104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35215 every 8 connection(s)
I20260926 21:41:36.091274 18039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:36.092758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 18039
I20260926 21:41:36.092962 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:41:36.093477 388 external_mini_cluster.cc:1428] Setting key 1fc2458d5a2c5bac5aebd6c9449776aa
I20260926 21:41:36.095679 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:36.109292 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:36.095458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.0.97.62:35215)
0926 21:41:36.096583 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:36.096598 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:36.096649 (+ 51us) server_negotiation.cc:407] Connection header received
0926 21:41:36.098853 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:36.098881 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:36.099519 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:36.099973 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:36.100530 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:36.102641 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:36.103933 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:36.104932 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:36.105124 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:36.106313 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:36.106328 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:36.106346 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:36.107126 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:36.107235 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:36.107271 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:36.107401 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:36.107582 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 21:41:36.107938 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":377,"threads_started":1}
I20260926 21:41:36.110457 18105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509: Bootstrap starting.
I20260926 21:41:36.113662 18105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:36.114663 18105 log.cc:824] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:36.117683 18105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509: No bootstrap required, opened a new log
I20260926 21:41:36.125389 18105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22935ee938c04aa192568aca46eca509" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35215 } }
I20260926 21:41:36.125792 18105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:36.125906 18105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22935ee938c04aa192568aca46eca509, State: Initialized, Role: FOLLOWER
I20260926 21:41:36.126495 18105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22935ee938c04aa192568aca46eca509" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35215 } }
I20260926 21:41:36.126729 18105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:36.126880 18105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:36.127046 18105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:36.129612 18105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22935ee938c04aa192568aca46eca509" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35215 } }
I20260926 21:41:36.130112 18105 leader_election.cc:302] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22935ee938c04aa192568aca46eca509; no voters:
I20260926 21:41:36.130707 18105 leader_election.cc:288] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:41:36.130865 18110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:36.131619 18110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [term 1 LEADER]: Becoming Leader. State: Replica: 22935ee938c04aa192568aca46eca509, State: Running, Role: LEADER
I20260926 21:41:36.132225 18110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22935ee938c04aa192568aca46eca509" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35215 } }
I20260926 21:41:36.132920 18105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:36.134934 18112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22935ee938c04aa192568aca46eca509. Latest consensus state: current_term: 1 leader_uuid: "22935ee938c04aa192568aca46eca509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22935ee938c04aa192568aca46eca509" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35215 } } }
I20260926 21:41:36.135357 18112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:36.135254 18111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22935ee938c04aa192568aca46eca509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22935ee938c04aa192568aca46eca509" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35215 } } }
I20260926 21:41:36.135555 18111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:36.137302 18118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:36.141819 18118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:36.150415 18118 catalog_manager.cc:1347] Generated new cluster ID: 4e8b07383a734f2291b32d5dcc34ee59
I20260926 21:41:36.150519 18118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:36.169973 18118 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:41:36.171754 18118 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:36.181313 18118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509: Generated new TSK 0
I20260926 21:41:36.182314 18118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:41:36.191473 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35215
--builtin_ntp_servers=127.0.97.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:41:36.528136 18129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:36.528457 18129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:36.528549 18129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:36.535969 18129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:36.536118 18129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:36.536273 18129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:41:36.545773 18129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:36.547024 18129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:36.548811 18129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:36.562115 18134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:36.562165 18135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:36.562198 18137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:36.563337 18129 server_base.cc:1034] running on GCE node
I20260926 21:41:36.563886 18129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:36.564810 18129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:36.566026 18129 hybrid_clock.cc:648] HybridClock initialized: now 1790458896565962 us; error 53 us; skew 500 ppm
I20260926 21:41:36.566344 18129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:36.568542 18129 webserver.cc:460] Webserver started at http://127.0.97.1:38019/ using document root <none> and password file <none>
I20260926 21:41:36.569275 18129 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:36.569401 18129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:36.569730 18129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:36.572587 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "c2946d2c60d348b580a6b6fce3b84f93"
format_stamp: "Formatted at 2026-09-26 21:41:36 on dist-test-slave-lhhq"
server_key: "65068adff3e46b146b5c94615bd60977"
server_key_iv: "cc0072b45d70c1c46df71dd6d8b1c74b"
server_key_version: "encryptionkey@0"
I20260926 21:41:36.573431 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c2946d2c60d348b580a6b6fce3b84f93"
format_stamp: "Formatted at 2026-09-26 21:41:36 on dist-test-slave-lhhq"
server_key: "65068adff3e46b146b5c94615bd60977"
server_key_iv: "cc0072b45d70c1c46df71dd6d8b1c74b"
server_key_version: "encryptionkey@0"
I20260926 21:41:36.578848 18129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:41:36.582186 18143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:36.583407 18129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:41:36.583591 18129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "c2946d2c60d348b580a6b6fce3b84f93"
format_stamp: "Formatted at 2026-09-26 21:41:36 on dist-test-slave-lhhq"
server_key: "65068adff3e46b146b5c94615bd60977"
server_key_iv: "cc0072b45d70c1c46df71dd6d8b1c74b"
server_key_version: "encryptionkey@0"
I20260926 21:41:36.583809 18129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:36.601924 18129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:36.602771 18129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:36.603056 18129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:36.604025 18129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:36.605602 18129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:36.605719 18129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:36.605845 18129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:36.605943 18129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:36.641443 18129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38853
I20260926 21:41:36.641520 18255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38853 every 8 connection(s)
I20260926 21:41:36.642814 18129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:41:36.646466 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 18129
I20260926 21:41:36.646665 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:41:36.647222 388 external_mini_cluster.cc:1428] Setting key 4f2ca0f5d9ce413e4176be4b71fc235d
I20260926 21:41:36.650870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35215
--builtin_ntp_servers=127.0.97.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:41:36.658735 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:36.645547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60357 (local address 127.0.97.62:35215)
0926 21:41:36.645995 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:36.646002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:36.647719 (+ 1717us) server_negotiation.cc:407] Connection header received
0926 21:41:36.650298 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:36.650303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:36.650375 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:36.650466 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:36.652075 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:36.652842 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:36.655583 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:36.656016 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:36.656117 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:36.657954 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:36.657959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:36.657962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:36.657981 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:36.658020 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:36.658032 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:36.658102 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:36.658378 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 21:41:36.658471 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":202,"threads_started":1}
I20260926 21:41:36.660244 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:36.646249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35215 (local address 127.0.97.1:60357)
0926 21:41:36.647501 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:36.647577 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:36.649970 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:36.650778 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:36.650794 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:36.650950 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:36.651860 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:36.651904 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:36.652997 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:36.653006 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:36.655388 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:36.655415 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:36.656269 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:36.656280 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:36.656653 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:36.657549 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:41:36.657555 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:36.657615 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:41:36.657736 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:36.658131 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:36.658146 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:36.658159 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 21:41:36.658353 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 21:41:36.658827 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":370,"threads_started":1}
I20260926 21:41:36.661981 18256 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35215
I20260926 21:41:36.662307 18256 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:36.663201 18256 heartbeater.cc:511] Master 127.0.97.62:35215 requested a full tablet report, sending...
I20260926 21:41:36.665597 18070 ts_manager.cc:194] Registered new tserver with Master: c2946d2c60d348b580a6b6fce3b84f93 (127.0.97.1:38853)
I20260926 21:41:36.667140 18070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:60357
W20260926 21:41:36.985911 18260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:36.986222 18260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:36.986369 18260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:36.993906 18260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:36.994015 18260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:36.994182 18260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:41:37.003288 18260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:37.004631 18260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:37.006474 18260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:37.019893 18265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:37.019929 18268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:37.019853 18266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:37.020743 18260 server_base.cc:1034] running on GCE node
I20260926 21:41:37.021561 18260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:37.022598 18260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:37.023840 18260 hybrid_clock.cc:648] HybridClock initialized: now 1790458897023785 us; error 53 us; skew 500 ppm
I20260926 21:41:37.024194 18260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:37.026651 18260 webserver.cc:460] Webserver started at http://127.0.97.2:45365/ using document root <none> and password file <none>
I20260926 21:41:37.027393 18260 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:37.027534 18260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:37.027859 18260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:37.031052 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "88ef3337a5c046b4aa64075fe53f792f"
format_stamp: "Formatted at 2026-09-26 21:41:37 on dist-test-slave-lhhq"
server_key: "c0b99501d746ffb88168908112d9af72"
server_key_iv: "ecb5a0f2cf528faf36b558be50539eec"
server_key_version: "encryptionkey@0"
I20260926 21:41:37.031894 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "88ef3337a5c046b4aa64075fe53f792f"
format_stamp: "Formatted at 2026-09-26 21:41:37 on dist-test-slave-lhhq"
server_key: "c0b99501d746ffb88168908112d9af72"
server_key_iv: "ecb5a0f2cf528faf36b558be50539eec"
server_key_version: "encryptionkey@0"
I20260926 21:41:37.037412 18260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 21:41:37.041102 18274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:37.042606 18260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:41:37.042834 18260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "88ef3337a5c046b4aa64075fe53f792f"
format_stamp: "Formatted at 2026-09-26 21:41:37 on dist-test-slave-lhhq"
server_key: "c0b99501d746ffb88168908112d9af72"
server_key_iv: "ecb5a0f2cf528faf36b558be50539eec"
server_key_version: "encryptionkey@0"
I20260926 21:41:37.043092 18260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:37.058991 18260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:37.059872 18260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:37.060173 18260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:37.061156 18260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:37.062708 18260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:37.062830 18260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:37.062975 18260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:37.063068 18260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:37.097949 18260 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36273
I20260926 21:41:37.097968 18386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36273 every 8 connection(s)
I20260926 21:41:37.099306 18260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:41:37.100148 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 18260
I20260926 21:41:37.100347 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:41:37.101014 388 external_mini_cluster.cc:1428] Setting key ea93bf2bfd6cd592ab42baab38f38558
I20260926 21:41:37.104882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35215
--builtin_ntp_servers=127.0.97.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:41:37.114394 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.102420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37303 (local address 127.0.97.62:35215)
0926 21:41:37.102647 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.102654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.104513 (+ 1859us) server_negotiation.cc:407] Connection header received
0926 21:41:37.107133 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.107141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.107233 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.107347 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:37.108642 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.109870 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.111444 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.112102 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.112248 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.113671 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:37.113676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:37.113680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:37.113699 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:37.113740 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:37.113753 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:37.113829 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.114087 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.114166 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:41:37.115906 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.103083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35215 (local address 127.0.97.2:37303)
0926 21:41:37.104360 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:37.104411 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:37.106793 (+ 2382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:37.107601 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:37.107613 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:37.107732 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:37.108465 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.108483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.110020 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.110024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.111282 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.111293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.112341 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.112349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.112583 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.113313 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:41:37.113319 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:37.113372 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:41:37.113476 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:37.113855 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:37.113867 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:37.113878 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 21:41:37.114071 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 21:41:37.114500 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":524,"threads_started":1}
I20260926 21:41:37.117420 18387 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35215
I20260926 21:41:37.117739 18387 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:37.118700 18387 heartbeater.cc:511] Master 127.0.97.62:35215 requested a full tablet report, sending...
I20260926 21:41:37.120289 18070 ts_manager.cc:194] Registered new tserver with Master: 88ef3337a5c046b4aa64075fe53f792f (127.0.97.2:36273)
I20260926 21:41:37.121575 18070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37303
W20260926 21:41:37.439405 18390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:37.439707 18390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:37.439807 18390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:37.447381 18390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:37.447546 18390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:37.447666 18390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:41:37.457167 18390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:37.458540 18390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:37.460274 18390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:37.474462 18396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:37.474538 18398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:37.474468 18395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:37.475575 18390 server_base.cc:1034] running on GCE node
I20260926 21:41:37.476217 18390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:37.477283 18390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:37.478515 18390 hybrid_clock.cc:648] HybridClock initialized: now 1790458897478455 us; error 68 us; skew 500 ppm
I20260926 21:41:37.478840 18390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:37.481120 18390 webserver.cc:460] Webserver started at http://127.0.97.3:36391/ using document root <none> and password file <none>
I20260926 21:41:37.481891 18390 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:37.482029 18390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:37.482359 18390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:37.485422 18390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "4a1f8b5220d44d8ba34dfda22870f6dd"
format_stamp: "Formatted at 2026-09-26 21:41:37 on dist-test-slave-lhhq"
server_key: "4cbe42dae402933b811887800735cc25"
server_key_iv: "a53398ab75a991a86103ed838ed7398b"
server_key_version: "encryptionkey@0"
I20260926 21:41:37.486250 18390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4a1f8b5220d44d8ba34dfda22870f6dd"
format_stamp: "Formatted at 2026-09-26 21:41:37 on dist-test-slave-lhhq"
server_key: "4cbe42dae402933b811887800735cc25"
server_key_iv: "a53398ab75a991a86103ed838ed7398b"
server_key_version: "encryptionkey@0"
I20260926 21:41:37.491828 18390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:41:37.495415 18404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:37.496869 18390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:41:37.497077 18390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "4a1f8b5220d44d8ba34dfda22870f6dd"
format_stamp: "Formatted at 2026-09-26 21:41:37 on dist-test-slave-lhhq"
server_key: "4cbe42dae402933b811887800735cc25"
server_key_iv: "a53398ab75a991a86103ed838ed7398b"
server_key_version: "encryptionkey@0"
I20260926 21:41:37.497392 18390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:37.514343 18390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:37.515232 18390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:37.515538 18390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:37.516552 18390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:37.518204 18390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:37.518337 18390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:37.518468 18390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:37.518558 18390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:37.553887 18390 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39527
I20260926 21:41:37.553987 18516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39527 every 8 connection(s)
I20260926 21:41:37.555330 18390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:41:37.564025 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 18390
I20260926 21:41:37.564209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:41:37.564770 388 external_mini_cluster.cc:1428] Setting key 669468f0ce28b911ab32adaa2d1fe60f
I20260926 21:41:37.568902 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.558127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49617 (local address 127.0.97.62:35215)
0926 21:41:37.558308 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.558314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.559756 (+ 1442us) server_negotiation.cc:407] Connection header received
0926 21:41:37.561657 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.561662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.561737 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.561836 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:37.562987 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.563749 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.566006 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.566467 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.566613 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.568067 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:37.568073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:37.568076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:37.568093 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:37.568125 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:37.568137 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:37.568236 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.568467 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.568577 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:41:37.570191 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.558637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35215 (local address 127.0.97.3:49617)
0926 21:41:37.559588 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:37.559624 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:37.561349 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:37.562090 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:37.562098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:37.562193 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:37.562803 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.562820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.563938 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.563944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.565826 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.565840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.566724 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.566731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.566974 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.567721 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:41:37.567723 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:37.567772 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:41:37.567863 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:37.568235 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:37.568249 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:37.568259 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:41:37.568458 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 21:41:37.568896 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":296,"threads_started":1}
I20260926 21:41:37.571791 18517 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35215
I20260926 21:41:37.572180 18517 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:37.573074 18517 heartbeater.cc:511] Master 127.0.97.62:35215 requested a full tablet report, sending...
I20260926 21:41:37.574580 18070 ts_manager.cc:194] Registered new tserver with Master: 4a1f8b5220d44d8ba34dfda22870f6dd (127.0.97.3:39527)
I20260926 21:41:37.575409 18070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49617
I20260926 21:41:37.580361 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:41:37.590574 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.584589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.0.97.62:35215)
0926 21:41:37.584859 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.584865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.585139 (+ 274us) server_negotiation.cc:407] Connection header received
0926 21:41:37.585383 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.585387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.585467 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.585560 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:37.586041 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.586726 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.588298 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.588731 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.588823 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.589879 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:37.589884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:37.589887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:37.589905 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:37.589956 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:37.589968 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:37.590037 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.590154 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.590231 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:41:37.594653 18070 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:41:37.601670 18070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:41:37.648622 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.615377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36273 (local address 127.0.0.1:41906)
0926 21:41:37.638298 (+ 22921us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:37.638333 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:37.638508 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:37.639635 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:37.639643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:37.639676 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:37.640231 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.640246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.642619 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.642624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.645249 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.645265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.647566 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.647573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.647799 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.647860 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 21:41:37.647994 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 21:41:37.648202 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22745,"mutex_wait_us":3,"thread_start_us":22575,"threads_started":1}
I20260926 21:41:37.649469 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.615416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.0.97.3:39527)
0926 21:41:37.639157 (+ 23741us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.639168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.639205 (+ 37us) server_negotiation.cc:407] Connection header received
0926 21:41:37.639313 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.639333 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.639612 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.640273 (+ 661us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:37.641013 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.642681 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.646220 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.647712 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.647844 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.648851 (+ 1007us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.648933 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.649094 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23561,"thread_start_us":23432,"threads_started":1}
I20260926 21:41:37.649570 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.616149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.0.97.2:36273)
0926 21:41:37.637766 (+ 21617us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.637780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.638482 (+ 702us) server_negotiation.cc:407] Connection header received
0926 21:41:37.638727 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.638741 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.639124 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.639360 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:37.640842 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.642429 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.645562 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.647041 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.647185 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.647445 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.649045 (+ 1600us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.649248 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21340,"thread_start_us":21205,"threads_started":1}
I20260926 21:41:37.649403 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.614597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.0.1:45414)
0926 21:41:37.630982 (+ 16385us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:37.631041 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:37.631317 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:37.640066 (+ 8749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:37.640072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:37.640104 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:37.640599 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.640613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.643301 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.643306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.646559 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.646572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.648653 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.648658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.648824 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.648855 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 21:41:37.648920 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 21:41:37.649057 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15456,"thread_start_us":15266,"threads_started":1}
I20260926 21:41:37.649792 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.615269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39527 (local address 127.0.0.1:46514)
0926 21:41:37.636019 (+ 20750us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:37.636039 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:37.636178 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:37.639858 (+ 3680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:37.639866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:37.639907 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:37.640695 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.640710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.642862 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.642867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.645996 (+ 3129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.646011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.648101 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.648107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.648286 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.648317 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 21:41:37.648397 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 21:41:37.648566 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20586,"mutex_wait_us":247,"thread_start_us":20414,"threads_started":1}
I20260926 21:41:37.651053 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.614800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.0.97.1:38853)
0926 21:41:37.639473 (+ 24673us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.639482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.639508 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:41:37.639606 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.639612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.639837 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.641324 (+ 1487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:37.641587 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.643134 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.646781 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.648441 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.649927 (+ 1486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.650541 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.650619 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.650790 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24512,"thread_start_us":25758,"threads_started":1}
I20260926 21:41:37.654196 18452 tablet_service.cc:1467] Processing CreateTablet for tablet 0c62b0d03c2d42d297decc689763d74a (DEFAULT_TABLE table=test-table [id=ce00004ad3fe496bbe90ae68bd5c6a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:37.654197 18322 tablet_service.cc:1467] Processing CreateTablet for tablet 0c62b0d03c2d42d297decc689763d74a (DEFAULT_TABLE table=test-table [id=ce00004ad3fe496bbe90ae68bd5c6a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:37.654157 18191 tablet_service.cc:1467] Processing CreateTablet for tablet 0c62b0d03c2d42d297decc689763d74a (DEFAULT_TABLE table=test-table [id=ce00004ad3fe496bbe90ae68bd5c6a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:37.655601 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c62b0d03c2d42d297decc689763d74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:37.655602 18322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c62b0d03c2d42d297decc689763d74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:37.655607 18191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c62b0d03c2d42d297decc689763d74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:37.666553 18535 tablet_bootstrap.cc:492] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: Bootstrap starting.
I20260926 21:41:37.670331 18536 tablet_bootstrap.cc:492] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: Bootstrap starting.
I20260926 21:41:37.671000 18256 heartbeater.cc:503] Master 127.0.97.62:35215 was elected leader, sending a full tablet report...
I20260926 21:41:37.672044 18537 tablet_bootstrap.cc:492] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: Bootstrap starting.
I20260926 21:41:37.674371 18536 tablet_bootstrap.cc:654] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:37.670995 18535 tablet_bootstrap.cc:654] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:37.675201 18537 tablet_bootstrap.cc:654] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:37.675683 18536 log.cc:824] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:37.676019 18535 log.cc:824] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:37.676393 18537 log.cc:824] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:37.680290 18536 tablet_bootstrap.cc:492] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: No bootstrap required, opened a new log
I20260926 21:41:37.680572 18536 ts_tablet_manager.cc:1397] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260926 21:41:37.680589 18535 tablet_bootstrap.cc:492] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: No bootstrap required, opened a new log
I20260926 21:41:37.680636 18537 tablet_bootstrap.cc:492] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: No bootstrap required, opened a new log
I20260926 21:41:37.680856 18535 ts_tablet_manager.cc:1397] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.003s
I20260926 21:41:37.680858 18537 ts_tablet_manager.cc:1397] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260926 21:41:37.688539 18536 raft_consensus.cc:348] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } }
I20260926 21:41:37.689071 18536 raft_consensus.cc:374] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:37.689174 18536 raft_consensus.cc:729] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1f8b5220d44d8ba34dfda22870f6dd, State: Initialized, Role: FOLLOWER
I20260926 21:41:37.689857 18536 consensus_queue.cc:260] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } }
I20260926 21:41:37.691044 18517 heartbeater.cc:503] Master 127.0.97.62:35215 was elected leader, sending a full tablet report...
I20260926 21:41:37.691596 18536 ts_tablet_manager.cc:1428] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: Time spent starting tablet: real 0.011s user 0.000s sys 0.012s
I20260926 21:41:37.692122 18535 raft_consensus.cc:348] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } }
I20260926 21:41:37.692571 18535 raft_consensus.cc:374] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:37.692365 18537 raft_consensus.cc:348] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } }
I20260926 21:41:37.692687 18535 raft_consensus.cc:729] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2946d2c60d348b580a6b6fce3b84f93, State: Initialized, Role: FOLLOWER
I20260926 21:41:37.692797 18537 raft_consensus.cc:374] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:37.692946 18537 raft_consensus.cc:729] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ef3337a5c046b4aa64075fe53f792f, State: Initialized, Role: FOLLOWER
I20260926 21:41:37.693465 18535 consensus_queue.cc:260] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } }
I20260926 21:41:37.693539 18537 consensus_queue.cc:260] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } }
I20260926 21:41:37.695341 18535 ts_tablet_manager.cc:1428] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260926 21:41:37.696163 18537 ts_tablet_manager.cc:1428] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260926 21:41:37.696638 18387 heartbeater.cc:503] Master 127.0.97.62:35215 was elected leader, sending a full tablet report...
I20260926 21:41:37.748013 18543 raft_consensus.cc:482] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:41:37.748378 18543 raft_consensus.cc:504] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } }
I20260926 21:41:37.750254 18543 leader_election.cc:288] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2946d2c60d348b580a6b6fce3b84f93 (127.0.97.1:38853), 4a1f8b5220d44d8ba34dfda22870f6dd (127.0.97.3:39527)
I20260926 21:41:37.757535 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.750630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55649 (local address 127.0.97.1:38853)
0926 21:41:37.750875 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.750884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.751786 (+ 902us) server_negotiation.cc:407] Connection header received
0926 21:41:37.751964 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.751968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.752040 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.752182 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:37.752983 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.753795 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.755764 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.756578 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.756738 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.756847 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.757123 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.757287 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 21:41:37.757560 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.750907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52591 (local address 127.0.97.3:39527)
0926 21:41:37.751147 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.751153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.751621 (+ 468us) server_negotiation.cc:407] Connection header received
0926 21:41:37.751943 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.751955 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.752017 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.752087 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:37.752736 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.753644 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.755799 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.756566 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.756700 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.756814 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.757153 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.757333 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 21:41:37.757658 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.750661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.97.2:55649)
0926 21:41:37.751662 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:37.751677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:37.751793 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:37.752274 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:37.752278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:37.752306 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:37.752785 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.752801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.753945 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.753949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.755539 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.755551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.756803 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.756809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.756961 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.756985 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 21:41:37.757060 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 21:41:37.757243 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":39,"thread_start_us":655,"threads_started":1}
I20260926 21:41:37.757721 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.750629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39527 (local address 127.0.97.2:52591)
0926 21:41:37.751478 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:37.751505 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:37.751757 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:37.752137 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:37.752144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:37.752172 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:37.752573 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.752582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.753778 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.753783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.755539 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:37.755551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.756772 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.756782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:37.756976 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.756998 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:41:37.757107 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 21:41:37.757286 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":95,"thread_start_us":449,"threads_started":1}
I20260926 21:41:37.758739 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c62b0d03c2d42d297decc689763d74a" candidate_uuid: "88ef3337a5c046b4aa64075fe53f792f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" is_pre_election: true
I20260926 21:41:37.758739 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c62b0d03c2d42d297decc689763d74a" candidate_uuid: "88ef3337a5c046b4aa64075fe53f792f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2946d2c60d348b580a6b6fce3b84f93" is_pre_election: true
I20260926 21:41:37.759262 18472 raft_consensus.cc:2454] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88ef3337a5c046b4aa64075fe53f792f in term 0.
I20260926 21:41:37.759264 18211 raft_consensus.cc:2454] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88ef3337a5c046b4aa64075fe53f792f in term 0.
I20260926 21:41:37.759984 18276 leader_election.cc:302] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a1f8b5220d44d8ba34dfda22870f6dd, 88ef3337a5c046b4aa64075fe53f792f, c2946d2c60d348b580a6b6fce3b84f93; no voters:
I20260926 21:41:37.760396 18543 raft_consensus.cc:2781] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:41:37.760556 18543 raft_consensus.cc:482] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:41:37.760666 18543 raft_consensus.cc:3037] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:37.763159 18543 raft_consensus.cc:504] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } }
I20260926 21:41:37.763818 18543 leader_election.cc:288] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [CANDIDATE]: Term 1 election: Requested vote from peers c2946d2c60d348b580a6b6fce3b84f93 (127.0.97.1:38853), 4a1f8b5220d44d8ba34dfda22870f6dd (127.0.97.3:39527)
I20260926 21:41:37.764285 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c62b0d03c2d42d297decc689763d74a" candidate_uuid: "88ef3337a5c046b4aa64075fe53f792f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2946d2c60d348b580a6b6fce3b84f93"
I20260926 21:41:37.764367 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c62b0d03c2d42d297decc689763d74a" candidate_uuid: "88ef3337a5c046b4aa64075fe53f792f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd"
I20260926 21:41:37.764565 18211 raft_consensus.cc:3037] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:37.764627 18472 raft_consensus.cc:3037] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:37.766923 18211 raft_consensus.cc:2454] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88ef3337a5c046b4aa64075fe53f792f in term 1.
I20260926 21:41:37.766927 18472 raft_consensus.cc:2454] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88ef3337a5c046b4aa64075fe53f792f in term 1.
I20260926 21:41:37.767459 18277 leader_election.cc:302] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a1f8b5220d44d8ba34dfda22870f6dd, 88ef3337a5c046b4aa64075fe53f792f, c2946d2c60d348b580a6b6fce3b84f93; no voters:
I20260926 21:41:37.767845 18543 raft_consensus.cc:2781] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:37.768306 18543 raft_consensus.cc:686] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 1 LEADER]: Becoming Leader. State: Replica: 88ef3337a5c046b4aa64075fe53f792f, State: Running, Role: LEADER
I20260926 21:41:37.768803 18543 consensus_queue.cc:237] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } }
I20260926 21:41:37.773638 18070 catalog_manager.cc:5495] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f reported cstate change: term changed from 0 to 1, leader changed from <none> to 88ef3337a5c046b4aa64075fe53f792f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "88ef3337a5c046b4aa64075fe53f792f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88ef3337a5c046b4aa64075fe53f792f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36273 } health_report { overall_health: HEALTHY } } }
I20260926 21:41:37.804412 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.798635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.0.97.2:36273)
0926 21:41:37.798842 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.798880 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.798904 (+ 24us) server_negotiation.cc:407] Connection header received
0926 21:41:37.799165 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.799169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.799262 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.799422 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:41:37.799806 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.800549 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.802162 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.802585 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.802728 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.803286 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:41:37.803804 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:41:37.803992 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.804042 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.804128 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 21:41:37.807075 18518 tablet.cc:2369] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:37.812682 18472 raft_consensus.cc:1261] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [term 1 FOLLOWER]: Refusing update from remote peer 88ef3337a5c046b4aa64075fe53f792f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:37.812731 18211 raft_consensus.cc:1261] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [term 1 FOLLOWER]: Refusing update from remote peer 88ef3337a5c046b4aa64075fe53f792f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:37.813784 18546 consensus_queue.cc:1035] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1f8b5220d44d8ba34dfda22870f6dd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:37.814141 18543 consensus_queue.cc:1035] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2946d2c60d348b580a6b6fce3b84f93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:37.826013 18549 mvcc.cc:205] Tried to move back new op lower bound from 7333719645431275520 to 7333719645265481728. Current Snapshot: MvccSnapshot[applied={T|T < 7333719645431275520}]
I20260926 21:41:37.840788 18069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:37.841053 18069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:37.844928 18069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22935ee938c04aa192568aca46eca509: Sending DeleteTablet for 3 replicas of tablet 0c62b0d03c2d42d297decc689763d74a
I20260926 21:41:37.846130 18191 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c62b0d03c2d42d297decc689763d74a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:37 UTC) from {username='slave'} at 127.0.0.1:45414
I20260926 21:41:37.846177 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c62b0d03c2d42d297decc689763d74a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:37 UTC) from {username='slave'} at 127.0.0.1:46514
I20260926 21:41:37.846925 18322 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c62b0d03c2d42d297decc689763d74a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:37 UTC) from {username='slave'} at 127.0.0.1:41906
I20260926 21:41:37.847364 18559 tablet_replica.cc:330] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: stopping tablet replica
I20260926 21:41:37.847592 18560 tablet_replica.cc:330] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: stopping tablet replica
I20260926 21:41:37.847944 18559 raft_consensus.cc:2229] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:37.847980 18560 raft_consensus.cc:2229] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:37.848457 18560 raft_consensus.cc:2258] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:37.848520 18559 raft_consensus.cc:2258] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:37.848867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 18129
I20260926 21:41:37.851485 18559 ts_tablet_manager.cc:1905] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:37.857694 18560 ts_tablet_manager.cc:1905] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:37.860927 18559 ts_tablet_manager.cc:1918] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:37.861207 18559 log.cc:1196] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/0c62b0d03c2d42d297decc689763d74a
I20260926 21:41:37.861872 18559 ts_tablet_manager.cc:1939] T 0c62b0d03c2d42d297decc689763d74a P 4a1f8b5220d44d8ba34dfda22870f6dd: Deleting consensus metadata
I20260926 21:41:37.863787 18055 catalog_manager.cc:4841] TS 4a1f8b5220d44d8ba34dfda22870f6dd (127.0.97.3:39527): tablet 0c62b0d03c2d42d297decc689763d74a (table test-table [id=ce00004ad3fe496bbe90ae68bd5c6a37]) successfully deleted
I20260926 21:41:37.864621 18561 tablet_replica.cc:330] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: stopping tablet replica
I20260926 21:41:37.865103 18561 raft_consensus.cc:2229] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:41:37.865793 18561 raft_consensus.cc:2258] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:37.868422 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:37.849959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.0.97.1:38853)
0926 21:41:37.850128 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:37.850136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:37.860145 (+ 10009us) server_negotiation.cc:407] Connection header received
0926 21:41:37.860615 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:37.860624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:37.860735 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:37.860823 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:37.861598 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.862675 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.864530 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:37.865014 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:37.865152 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:37.866445 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:37.866460 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:37.866473 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:37.867054 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:37.867101 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:37.867124 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:37.867194 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:37.867904 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 21:41:37.868025 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:41:37.868728 18561 ts_tablet_manager.cc:1905] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:37.869792 18251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:37.876835 18561 ts_tablet_manager.cc:1918] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:37.877027 18561 log.cc:1196] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/0c62b0d03c2d42d297decc689763d74a
I20260926 21:41:37.877560 18561 ts_tablet_manager.cc:1939] T 0c62b0d03c2d42d297decc689763d74a P 88ef3337a5c046b4aa64075fe53f792f: Deleting consensus metadata
I20260926 21:41:37.879045 18054 catalog_manager.cc:4841] TS 88ef3337a5c046b4aa64075fe53f792f (127.0.97.2:36273): tablet 0c62b0d03c2d42d297decc689763d74a (table test-table [id=ce00004ad3fe496bbe90ae68bd5c6a37]) successfully deleted
I20260926 21:41:37.983124 18560 ts_tablet_manager.cc:1918] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:37.983448 18560 log.cc:1196] T 0c62b0d03c2d42d297decc689763d74a P c2946d2c60d348b580a6b6fce3b84f93: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/0c62b0d03c2d42d297decc689763d74a
I20260926 21:41:37.983811 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 18129
I20260926 21:41:38.007967 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 18260
W20260926 21:41:38.008013 18056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38853 (error 108)
I20260926 21:41:38.010771 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.010268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.0.1:45432)
0926 21:41:38.010462 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:38.010591 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 21:41:38.011145 18056 catalog_manager.cc:4584] TS c2946d2c60d348b580a6b6fce3b84f93 (127.0.97.1:38853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c62b0d03c2d42d297decc689763d74a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38853: connect: Connection refused (error 111)
I20260926 21:41:38.016170 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.008820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.0.97.2:36273)
0926 21:41:38.009489 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:38.009495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:38.009522 (+ 27us) server_negotiation.cc:407] Connection header received
0926 21:41:38.009718 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:38.009723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:38.009829 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:38.009928 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:38.010583 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:38.011620 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:38.012944 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:38.013469 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:38.013614 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:38.014708 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:38.014717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:38.014731 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:38.015367 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:38.015415 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:38.015434 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:38.015572 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:38.015699 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 21:41:38.015803 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477}
I20260926 21:41:38.016924 18382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:38.064872 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.064392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.0.1:45444)
0926 21:41:38.064623 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:38.064715 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 21:41:38.065987 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.065665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.0.1:45452)
0926 21:41:38.065798 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:38.065867 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 21:41:38.126559 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.126030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.0.1:45454)
0926 21:41:38.126288 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:38.126384 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 21:41:38.127604 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.127316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.0.1:45458)
0926 21:41:38.127440 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:38.127505 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 21:41:38.137511 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 18260
I20260926 21:41:38.158654 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 18390
I20260926 21:41:38.165906 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.159445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.0.97.3:39527)
0926 21:41:38.159619 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:38.159624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:38.159696 (+ 72us) server_negotiation.cc:407] Connection header received
0926 21:41:38.159976 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:38.159981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:38.160080 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:38.160178 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:38.160642 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:38.161523 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:38.162790 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:38.163167 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:38.163260 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:38.164482 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:38.164491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:38.164506 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:38.165136 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:38.165252 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:38.165285 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:38.165380 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:38.165585 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 21:41:38.165670 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:41:38.166571 18512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:38.208700 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.208144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.0.1:45462)
0926 21:41:38.208392 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:38.208511 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 21:41:38.209862 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.209435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.0.1:45468)
0926 21:41:38.209645 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:38.209728 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 21:41:38.256476 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 18390
I20260926 21:41:38.275276 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 18039
I20260926 21:41:38.276402 18100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:38.386919 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:38.386198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38853 (local address 127.0.0.1:52802)
0926 21:41:38.386407 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:38.386578 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 21:41:38.387161 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 18039
2026-09-26T21:41:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2810 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:41:38 dist-test-slave-lhhq krb5kdc[18570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:41:38 dist-test-slave-lhhq krb5kdc[18570](info): set up 2 sockets
Sep 26 21:41:38 dist-test-slave-lhhq krb5kdc[18570](info): commencing operation
krb5kdc: starting...
W20260926 21:41:39.513394 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:41:39 dist-test-slave-lhhq krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:41:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:41:39.770171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37343
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:40.106280 18583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:40.106631 18583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:40.106801 18583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:40.114218 18583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:40.114362 18583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:40.114472 18583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:40.114590 18583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:40.114670 18583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:40.124686 18583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:40.126071 18583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:40.127506 18583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:40.139183 18588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:40.139087 18589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:40.139142 18591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:40.140041 18583 server_base.cc:1034] running on GCE node
I20260926 21:41:40.140813 18583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:40.142267 18583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:40.143496 18583 hybrid_clock.cc:648] HybridClock initialized: now 1790458900143446 us; error 59 us; skew 500 ppm
I20260926 21:41:40.143858 18583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:40 dist-test-slave-lhhq krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:40.156749 18583 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:41:40.158209 18583 webserver.cc:460] Webserver started at http://127.0.97.62:37627/ using document root <none> and password file <none>
I20260926 21:41:40.158859 18583 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:40.158954 18583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:40.159204 18583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:40.162117 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "46ba2cfc3e324601957b2ea2a4c92c55"
format_stamp: "Formatted at 2026-09-26 21:41:40 on dist-test-slave-lhhq"
server_key: "71c24902db5414cbd625d0d95461da90"
server_key_iv: "6932e5725cf10a1e1072196c1a63e99a"
server_key_version: "encryptionkey@0"
I20260926 21:41:40.162791 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "46ba2cfc3e324601957b2ea2a4c92c55"
format_stamp: "Formatted at 2026-09-26 21:41:40 on dist-test-slave-lhhq"
server_key: "71c24902db5414cbd625d0d95461da90"
server_key_iv: "6932e5725cf10a1e1072196c1a63e99a"
server_key_version: "encryptionkey@0"
I20260926 21:41:40.167822 18583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:41:40.171370 18598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:40.172691 18583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:41:40.172899 18583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "46ba2cfc3e324601957b2ea2a4c92c55"
format_stamp: "Formatted at 2026-09-26 21:41:40 on dist-test-slave-lhhq"
server_key: "71c24902db5414cbd625d0d95461da90"
server_key_iv: "6932e5725cf10a1e1072196c1a63e99a"
server_key_version: "encryptionkey@0"
I20260926 21:41:40.173089 18583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:40.190985 18583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:40.195365 18583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:40.195627 18583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:40.214051 18583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37343
I20260926 21:41:40.214083 18649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37343 every 8 connection(s)
I20260926 21:41:40.215409 18583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:40.219388 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:40.225764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 18583
I20260926 21:41:40.225955 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:41:40.226406 388 external_mini_cluster.cc:1428] Setting key 5be86328f17e3ee1fc0ffaf37e4bf0ba
I20260926 21:41:40.230826 18650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55: Bootstrap starting.
I20260926 21:41:40.234542 18650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:40.235826 18650 log.cc:824] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:40.239007 18650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55: No bootstrap required, opened a new log
Sep 26 21:41:40 dist-test-slave-lhhq krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:40.247804 18650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ba2cfc3e324601957b2ea2a4c92c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37343 } }
I20260926 21:41:40.248216 18650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:40.248353 18650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46ba2cfc3e324601957b2ea2a4c92c55, State: Initialized, Role: FOLLOWER
I20260926 21:41:40.249083 18650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ba2cfc3e324601957b2ea2a4c92c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37343 } }
I20260926 21:41:40.249430 18650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:40.249584 18650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:40.249827 18650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:40.250617 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:40.228356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.0.97.62:37343)
0926 21:41:40.229231 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:40.229248 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:40.229299 (+ 51us) server_negotiation.cc:407] Connection header received
0926 21:41:40.231018 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:40.231051 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:40.231418 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:40.231730 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:40.233634 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:40.234985 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:40.236928 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:40.237688 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:40.237816 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:40.242741 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:40.242752 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:40.242768 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:40.242814 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:40.246067 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:40.246825 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:40.246832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:40.246839 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:40.247024 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:40.247418 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:40.247425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:40.247428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:40.247987 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:40.248288 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:40.248747 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 21:41:40.249241 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":416,"threads_started":1}
I20260926 21:41:40.252838 18650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ba2cfc3e324601957b2ea2a4c92c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37343 } }
I20260926 21:41:40.253434 18650 leader_election.cc:302] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46ba2cfc3e324601957b2ea2a4c92c55; no voters:
I20260926 21:41:40.254149 18650 leader_election.cc:288] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:41:40.254256 18655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:40.254894 18655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [term 1 LEADER]: Becoming Leader. State: Replica: 46ba2cfc3e324601957b2ea2a4c92c55, State: Running, Role: LEADER
I20260926 21:41:40.255518 18655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ba2cfc3e324601957b2ea2a4c92c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37343 } }
I20260926 21:41:40.256469 18650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:40.258983 18657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46ba2cfc3e324601957b2ea2a4c92c55. Latest consensus state: current_term: 1 leader_uuid: "46ba2cfc3e324601957b2ea2a4c92c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ba2cfc3e324601957b2ea2a4c92c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37343 } } }
I20260926 21:41:40.259229 18656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46ba2cfc3e324601957b2ea2a4c92c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ba2cfc3e324601957b2ea2a4c92c55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37343 } } }
I20260926 21:41:40.259487 18657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:40.259512 18656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:40.261025 18662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:40.266630 18662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:40.276407 18662 catalog_manager.cc:1347] Generated new cluster ID: 6c3c9bb4827b484d930b7b476ad53eda
I20260926 21:41:40.276571 18662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:40.298481 18662 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:41:40.300438 18662 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:40.309731 18662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55: Generated new TSK 0
I20260926 21:41:40.310832 18662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:41:40.373894 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37343
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:40.709878 18676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:40.710237 18676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:40.710419 18676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:40.718084 18676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:40.718246 18676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:40.718451 18676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:41:40.732985 18676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:40.734535 18676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:40.735912 18676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:40.749835 18682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:40.749989 18684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:40.749917 18681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:40.750929 18676 server_base.cc:1034] running on GCE node
I20260926 21:41:40.751647 18676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:40.752651 18676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:40.753939 18676 hybrid_clock.cc:648] HybridClock initialized: now 1790458900753854 us; error 61 us; skew 500 ppm
I20260926 21:41:40.754294 18676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:40 dist-test-slave-lhhq krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:40.757803 18676 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:41:40.759356 18676 webserver.cc:460] Webserver started at http://127.0.97.1:36121/ using document root <none> and password file <none>
I20260926 21:41:40.760072 18676 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:40.760224 18676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:40.760537 18676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:40.763682 18676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "dad0ba7f2f494984a3b167310151f610"
format_stamp: "Formatted at 2026-09-26 21:41:40 on dist-test-slave-lhhq"
server_key: "a76219cc8b3f399d9748a2b83fe465ce"
server_key_iv: "f3f5bd11f1e1759acb1bf22319809d25"
server_key_version: "encryptionkey@0"
I20260926 21:41:40.764487 18676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dad0ba7f2f494984a3b167310151f610"
format_stamp: "Formatted at 2026-09-26 21:41:40 on dist-test-slave-lhhq"
server_key: "a76219cc8b3f399d9748a2b83fe465ce"
server_key_iv: "f3f5bd11f1e1759acb1bf22319809d25"
server_key_version: "encryptionkey@0"
I20260926 21:41:40.769744 18676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:41:40.773365 18691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:40.774813 18676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:41:40.775017 18676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "dad0ba7f2f494984a3b167310151f610"
format_stamp: "Formatted at 2026-09-26 21:41:40 on dist-test-slave-lhhq"
server_key: "a76219cc8b3f399d9748a2b83fe465ce"
server_key_iv: "f3f5bd11f1e1759acb1bf22319809d25"
server_key_version: "encryptionkey@0"
I20260926 21:41:40.775241 18676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:40.797726 18676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:40.802580 18676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:40.802904 18676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:40.803952 18676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:40.805465 18676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:40.805572 18676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:40.805725 18676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:40.805811 18676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:40.840197 18676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46065
I20260926 21:41:40.840265 18803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46065 every 8 connection(s)
I20260926 21:41:40.841665 18676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:41:40.851852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 18676
I20260926 21:41:40.852061 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:41:40.852505 388 external_mini_cluster.cc:1428] Setting key 8d4833e6a11513b7bd62889215ce4fe4
Sep 26 21:41:40 dist-test-slave-lhhq krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:40.861575 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:40.844328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47223 (local address 127.0.97.62:37343)
0926 21:41:40.844791 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:40.844796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:40.845682 (+ 886us) server_negotiation.cc:407] Connection header received
0926 21:41:40.847548 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:40.847553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:40.847642 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:40.847769 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:40.849267 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:40.850003 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:40.851528 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:40.851926 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:40.852072 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:40.855807 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:40.855811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:40.855815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:40.855845 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:40.858554 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:40.859258 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:40.859265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:40.859269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:40.859389 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:40.859821 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:40.859825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:40.859830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:40.860083 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:40.860236 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:40.860926 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 21:41:40.861152 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":256,"threads_started":1}
I20260926 21:41:40.862723 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:40.844758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37343 (local address 127.0.97.1:47223)
0926 21:41:40.845521 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:40.845566 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:40.847265 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:40.848013 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:40.848021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:40.848531 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:40.849084 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:40.849100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:40.850166 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:40.850170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:40.851378 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:40.851388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:40.852200 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:40.852206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:40.852423 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:40.853157 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:41:40.853257 (+ 100us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:40.855574 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:40.858689 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:40.858698 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:40.858714 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:40.859130 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:40.859501 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:40.859506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:40.859510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:40.859698 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:40.860212 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:40.860220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:40.860554 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 21:41:40.860818 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 21:41:40.861421 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":267,"threads_started":1}
I20260926 21:41:40.864352 18804 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37343
I20260926 21:41:40.864677 18804 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:40.865603 18804 heartbeater.cc:511] Master 127.0.97.62:37343 requested a full tablet report, sending...
I20260926 21:41:40.867837 18615 ts_manager.cc:194] Registered new tserver with Master: dad0ba7f2f494984a3b167310151f610 (127.0.97.1:46065)
I20260926 21:41:40.869344 18615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47223
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:41:40.907819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37343
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:41.236281 18810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:41.236544 18810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:41.236636 18810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:41.243896 18810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:41.244006 18810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:41.244127 18810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:41:41.253826 18810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:41.255158 18810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:41.256423 18810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:41.269589 18816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:41.269670 18815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:41.269618 18818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:41.270653 18810 server_base.cc:1034] running on GCE node
I20260926 21:41:41.271253 18810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:41.272177 18810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:41.273401 18810 hybrid_clock.cc:648] HybridClock initialized: now 1790458901273339 us; error 71 us; skew 500 ppm
I20260926 21:41:41.273711 18810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:41 dist-test-slave-lhhq krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:41.276799 18810 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:41:41.278378 18810 webserver.cc:460] Webserver started at http://127.0.97.2:41985/ using document root <none> and password file <none>
I20260926 21:41:41.279022 18810 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:41.279153 18810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:41.279486 18810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:41.282567 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489"
format_stamp: "Formatted at 2026-09-26 21:41:41 on dist-test-slave-lhhq"
server_key: "792f6d547b27cbcf5300a64081711275"
server_key_iv: "c3af01fa31f653c849a13ed4d691bea9"
server_key_version: "encryptionkey@0"
I20260926 21:41:41.283349 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489"
format_stamp: "Formatted at 2026-09-26 21:41:41 on dist-test-slave-lhhq"
server_key: "792f6d547b27cbcf5300a64081711275"
server_key_iv: "c3af01fa31f653c849a13ed4d691bea9"
server_key_version: "encryptionkey@0"
I20260926 21:41:41.288295 18810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:41:41.291704 18825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:41.292936 18810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:41:41.293092 18810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489"
format_stamp: "Formatted at 2026-09-26 21:41:41 on dist-test-slave-lhhq"
server_key: "792f6d547b27cbcf5300a64081711275"
server_key_iv: "c3af01fa31f653c849a13ed4d691bea9"
server_key_version: "encryptionkey@0"
I20260926 21:41:41.293318 18810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:41.336462 18810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:41.341321 18810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:41.341658 18810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:41.342662 18810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:41.344070 18810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:41.344144 18810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:41.344300 18810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:41.344375 18810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:41.378396 18810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40907
I20260926 21:41:41.378443 18937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40907 every 8 connection(s)
I20260926 21:41:41.379791 18810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:41:41.381640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 18810
I20260926 21:41:41.381785 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:41:41.382182 388 external_mini_cluster.cc:1428] Setting key 5305477e510de1e5792a8c6aab5b385f
Sep 26 21:41:41 dist-test-slave-lhhq krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:41.404163 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.383959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42559 (local address 127.0.97.62:37343)
0926 21:41:41.384411 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:41.384423 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:41.385696 (+ 1273us) server_negotiation.cc:407] Connection header received
0926 21:41:41.388645 (+ 2949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:41.388649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:41.388716 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:41.388809 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:41.391063 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.391792 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.393448 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.393874 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.394013 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:41.398121 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:41.398126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:41.398129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:41.398165 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:41.400813 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:41.401722 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:41.401734 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:41.401737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:41.401878 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:41.402336 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:41.402345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:41.402347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:41.402628 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:41.402793 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:41.403524 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 21:41:41.403763 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":237,"threads_started":1}
I20260926 21:41:41.405270 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.384626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37343 (local address 127.0.97.2:42559)
0926 21:41:41.385514 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:41.385571 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:41.388342 (+ 2771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:41.389104 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:41.389120 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:41.389944 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:41.390787 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:41.390817 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.391972 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.391978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:41.393294 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:41.393307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.394107 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.394114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:41.394477 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:41.395300 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:41:41.395346 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:41.397855 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:41.400995 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:41.401005 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:41.401024 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:41.401580 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:41.401996 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:41.402001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:41.402005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:41.402207 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:41.402764 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:41.402772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:41.403136 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 21:41:41.403419 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 21:41:41.403990 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":325,"threads_started":1}
I20260926 21:41:41.406836 18938 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37343
I20260926 21:41:41.407212 18938 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:41.408185 18938 heartbeater.cc:511] Master 127.0.97.62:37343 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 21:41:41.409966 18615 ts_manager.cc:194] Registered new tserver with Master: 6a8cafdd2e1a474a8d9c58e8dcb8f489 (127.0.97.2:40907)
I20260926 21:41:41.410821 18615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:41:41.440649 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37343
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:41.780701 18944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:41.781006 18944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:41.781158 18944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:41.788730 18944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:41.788858 18944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:41.789039 18944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:41:41.799916 18944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:41.801378 18944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:41.802722 18944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:41.815897 18949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:41.815937 18950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:41.815969 18952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:41.817350 18944 server_base.cc:1034] running on GCE node
I20260926 21:41:41.817962 18944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:41.818890 18944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:41.820101 18944 hybrid_clock.cc:648] HybridClock initialized: now 1790458901820058 us; error 34 us; skew 500 ppm
I20260926 21:41:41.820397 18944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:41 dist-test-slave-lhhq krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:41.823645 18944 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:41:41.825228 18944 webserver.cc:460] Webserver started at http://127.0.97.3:42689/ using document root <none> and password file <none>
I20260926 21:41:41.825958 18944 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:41.826102 18944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:41.826414 18944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:41.829345 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "cd136b80b6884be2851f12aea2d0acdf"
format_stamp: "Formatted at 2026-09-26 21:41:41 on dist-test-slave-lhhq"
server_key: "6a56cb03dab889e33b361a3f7e354907"
server_key_iv: "6017623df07ad3e363b03ffc44b3607b"
server_key_version: "encryptionkey@0"
I20260926 21:41:41.830123 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cd136b80b6884be2851f12aea2d0acdf"
format_stamp: "Formatted at 2026-09-26 21:41:41 on dist-test-slave-lhhq"
server_key: "6a56cb03dab889e33b361a3f7e354907"
server_key_iv: "6017623df07ad3e363b03ffc44b3607b"
server_key_version: "encryptionkey@0"
I20260926 21:41:41.835127 18944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:41:41.838575 18959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:41.840232 18944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:41:41.840402 18944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "cd136b80b6884be2851f12aea2d0acdf"
format_stamp: "Formatted at 2026-09-26 21:41:41 on dist-test-slave-lhhq"
server_key: "6a56cb03dab889e33b361a3f7e354907"
server_key_iv: "6017623df07ad3e363b03ffc44b3607b"
server_key_version: "encryptionkey@0"
I20260926 21:41:41.840638 18944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:41.873013 18804 heartbeater.cc:503] Master 127.0.97.62:37343 was elected leader, sending a full tablet report...
I20260926 21:41:41.876649 18944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:41.881304 18944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:41.881650 18944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:41.882670 18944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:41.883924 18944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:41.883996 18944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:41.884087 18944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:41.884132 18944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:41.917721 18944 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38901
I20260926 21:41:41.917807 19071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38901 every 8 connection(s)
I20260926 21:41:41.918998 18944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:41:41.926018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 18944
I20260926 21:41:41.926187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:41:41.926689 388 external_mini_cluster.cc:1428] Setting key 407ce129f092a3c9111c3015541f632d
Sep 26 21:41:41 dist-test-slave-lhhq krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:41.939759 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.921586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37443 (local address 127.0.97.62:37343)
0926 21:41:41.922098 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:41.922102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:41.922954 (+ 852us) server_negotiation.cc:407] Connection header received
0926 21:41:41.924842 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:41.924849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:41.924960 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:41.925083 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:41.926773 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.927828 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.930018 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.930417 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.930520 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:41.934385 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:41.934390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:41.934394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:41.934447 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:41.936965 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:41.937768 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:41.937772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:41.937774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:41.937889 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:41.938276 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:41.938280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:41.938281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:41.938550 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:41.938668 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:41.939316 (+ 648us) server_negotiation.cc:299] Negotiation successful
0926 21:41:41.939497 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
I20260926 21:41:41.940830 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.922104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37343 (local address 127.0.97.3:37443)
0926 21:41:41.922776 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:41.922815 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:41.924545 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:41.925362 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:41.925371 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:41.925900 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:41.926585 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:41.926603 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.928022 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.928028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:41.929857 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:41.929877 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.930654 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.930661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:41.930892 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:41.931652 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:41:41.931679 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:41.934061 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:41.937129 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:41.937139 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:41.937160 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:41.937648 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:41.937988 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:41.937991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:41.937994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:41.938177 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:41.938654 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:41.938660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:41.938991 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 21:41:41.939221 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 21:41:41.939612 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":282,"threads_started":1}
I20260926 21:41:41.942414 19072 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37343
I20260926 21:41:41.942726 19072 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:41.943456 19072 heartbeater.cc:511] Master 127.0.97.62:37343 requested a full tablet report, sending...
I20260926 21:41:41.944887 18614 ts_manager.cc:194] Registered new tserver with Master: cd136b80b6884be2851f12aea2d0acdf (127.0.97.3:38901)
I20260926 21:41:41.945974 18614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37443
I20260926 21:41:41.955065 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:41:41.972671 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.959369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.0.97.62:37343)
0926 21:41:41.959654 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:41.959661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:41.959991 (+ 330us) server_negotiation.cc:407] Connection header received
0926 21:41:41.960195 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:41.960200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:41.960261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:41.960374 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:41.961895 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.962704 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.964015 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:41.964455 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:41.964593 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:41.967348 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:41.967353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:41.967357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:41.967389 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:41.969884 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:41.970650 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:41.970657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:41.970665 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:41.970779 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:41.971220 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:41.971232 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:41.971235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:41.971536 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:41.971822 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:41.972183 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 21:41:41.972394 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 21:41:41.976804 18614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:41:41.983902 18614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:41:42.007920 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.996486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.0.97.1:46065)
0926 21:41:41.997792 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:41.997805 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:41.998496 (+ 691us) server_negotiation.cc:407] Connection header received
0926 21:41:41.998902 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:41.998910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:41.999201 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:41.999374 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:42.000352 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.002049 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.004900 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.006394 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.006497 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.006729 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:42.007373 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 21:41:42.007555 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076,"thread_start_us":811,"threads_started":1}
I20260926 21:41:42.008841 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.996039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46065 (local address 127.0.0.1:56252)
0926 21:41:41.998338 (+ 2299us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:41.998382 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:41.998642 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:41.999720 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:41.999726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:41.999753 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:42.000174 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.000187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.002229 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.002234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.004712 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.004727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.006906 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.006912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.007095 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.007158 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 21:41:42.008279 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0926 21:41:42.008496 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2074,"thread_start_us":1945,"threads_started":1}
I20260926 21:41:42.010272 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.996196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40907 (local address 127.0.0.1:49826)
0926 21:41:41.999413 (+ 3217us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:41.999432 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:41.999543 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:42.000377 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:42.000383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:42.000411 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:42.000854 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.000867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.002792 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.002796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.006060 (+ 3264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.006075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.009492 (+ 3417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.009502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.009687 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.009719 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 21:41:42.009806 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 21:41:42.009947 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3099,"thread_start_us":2952,"threads_started":1}
I20260926 21:41:42.010957 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.997010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.0.97.2:40907)
0926 21:41:41.999004 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:41.999013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:41.999686 (+ 673us) server_negotiation.cc:407] Connection header received
0926 21:41:41.999790 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:41.999800 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:42.000044 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:42.000221 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:42.001072 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.002628 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.006258 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.008963 (+ 2705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.009092 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.009328 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:42.010508 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0926 21:41:42.010692 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1834,"thread_start_us":1697,"threads_started":1}
I20260926 21:41:42.013098 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce408d0b91d46508252ba89d6bbe133 (DEFAULT_TABLE table=test-table [id=b70f46d132454391bd48da6c5b071858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:42.014685 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce408d0b91d46508252ba89d6bbe133. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:42.014647 18873 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce408d0b91d46508252ba89d6bbe133 (DEFAULT_TABLE table=test-table [id=b70f46d132454391bd48da6c5b071858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:42.015980 18873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce408d0b91d46508252ba89d6bbe133. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:42.016659 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:41.999558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38901 (local address 127.0.0.1:44892)
0926 21:41:42.009687 (+ 10129us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:42.009706 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:42.009835 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:42.010768 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:42.010774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:42.010803 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:42.011137 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.011149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.013053 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.013060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.014472 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.014484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.015889 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.015895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.016066 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.016102 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 21:41:42.016205 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 21:41:42.016327 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9982,"thread_start_us":9825,"threads_started":1}
I20260926 21:41:42.016763 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:42.000162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.0.97.3:38901)
0926 21:41:42.008506 (+ 8344us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:42.008513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:42.009962 (+ 1449us) server_negotiation.cc:407] Connection header received
0926 21:41:42.010083 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:42.010090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:42.010318 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:42.010519 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:42.011328 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.012899 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.014616 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.015674 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.015815 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.016030 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:42.016257 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 21:41:42.016419 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8177,"thread_start_us":8016,"threads_started":1}
I20260926 21:41:42.019675 19007 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce408d0b91d46508252ba89d6bbe133 (DEFAULT_TABLE table=test-table [id=b70f46d132454391bd48da6c5b071858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:42.021006 19007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce408d0b91d46508252ba89d6bbe133. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:42.029837 19091 tablet_bootstrap.cc:492] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: Bootstrap starting.
I20260926 21:41:42.032327 19093 tablet_bootstrap.cc:492] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: Bootstrap starting.
I20260926 21:41:42.033129 19092 tablet_bootstrap.cc:492] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: Bootstrap starting.
I20260926 21:41:42.034401 19091 tablet_bootstrap.cc:654] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:42.035519 19093 tablet_bootstrap.cc:654] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:42.036031 19091 log.cc:824] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:42.036544 19092 tablet_bootstrap.cc:654] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:42.036765 19093 log.cc:824] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:42.037953 19092 log.cc:824] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:42.039847 19091 tablet_bootstrap.cc:492] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: No bootstrap required, opened a new log
I20260926 21:41:42.039844 19093 tablet_bootstrap.cc:492] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: No bootstrap required, opened a new log
I20260926 21:41:42.040122 19091 ts_tablet_manager.cc:1397] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260926 21:41:42.040122 19093 ts_tablet_manager.cc:1397] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 21:41:42.040515 19092 tablet_bootstrap.cc:492] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: No bootstrap required, opened a new log
I20260926 21:41:42.040759 19092 ts_tablet_manager.cc:1397] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 21:41:42.048276 19092 raft_consensus.cc:348] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } }
I20260926 21:41:42.048664 19092 raft_consensus.cc:374] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:42.048765 19092 raft_consensus.cc:729] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8cafdd2e1a474a8d9c58e8dcb8f489, State: Initialized, Role: FOLLOWER
I20260926 21:41:42.049425 19092 consensus_queue.cc:260] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } }
I20260926 21:41:42.050506 18938 heartbeater.cc:503] Master 127.0.97.62:37343 was elected leader, sending a full tablet report...
I20260926 21:41:42.051234 19092 ts_tablet_manager.cc:1428] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260926 21:41:42.051057 19093 raft_consensus.cc:348] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } }
I20260926 21:41:42.051057 19091 raft_consensus.cc:348] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } }
I20260926 21:41:42.051465 19091 raft_consensus.cc:374] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:42.051461 19093 raft_consensus.cc:374] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:42.051556 19093 raft_consensus.cc:729] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd136b80b6884be2851f12aea2d0acdf, State: Initialized, Role: FOLLOWER
I20260926 21:41:42.051556 19091 raft_consensus.cc:729] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad0ba7f2f494984a3b167310151f610, State: Initialized, Role: FOLLOWER
I20260926 21:41:42.052146 19091 consensus_queue.cc:260] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } }
I20260926 21:41:42.052148 19093 consensus_queue.cc:260] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } }
I20260926 21:41:42.053730 19072 heartbeater.cc:503] Master 127.0.97.62:37343 was elected leader, sending a full tablet report...
I20260926 21:41:42.054301 19093 ts_tablet_manager.cc:1428] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260926 21:41:42.054859 19091 ts_tablet_manager.cc:1428] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
W20260926 21:41:42.094349 18805 tablet.cc:2369] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:41:42.131857 18939 tablet.cc:2369] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:41:42.170666 19073 tablet.cc:2369] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:42.182720 19099 raft_consensus.cc:482] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:41:42.183089 19099 raft_consensus.cc:504] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } }
I20260926 21:41:42.184857 19099 leader_election.cc:288] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dad0ba7f2f494984a3b167310151f610 (127.0.97.1:46065), 6a8cafdd2e1a474a8d9c58e8dcb8f489 (127.0.97.2:40907)
I20260926 21:41:42.191004 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:42.185280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60195 (local address 127.0.97.1:46065)
0926 21:41:42.185490 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:42.185496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:42.185605 (+ 109us) server_negotiation.cc:407] Connection header received
0926 21:41:42.185835 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:42.185840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:42.185899 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:42.186002 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:42.186373 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.187113 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.188934 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.190161 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.190279 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.190358 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:42.190706 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 21:41:42.190803 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 21:41:42.191109 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:42.185257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46065 (local address 127.0.97.3:60195)
0926 21:41:42.185499 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:42.185519 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:42.185681 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:42.186030 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:42.186034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:42.186061 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:42.186241 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.186250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.187265 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.187270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.188751 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.188768 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.190324 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.190331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.190531 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.190573 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 21:41:42.190678 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 21:41:42.190811 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1}
I20260926 21:41:42.192288 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:42.185361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52371 (local address 127.0.97.2:40907)
0926 21:41:42.185563 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:42.185571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:42.186266 (+ 695us) server_negotiation.cc:407] Connection header received
0926 21:41:42.186476 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:42.186480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:42.186538 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:42.186633 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:42.187117 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.187874 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.190200 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.191464 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.191595 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.191693 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:42.192023 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 21:41:42.192115 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:41:42.192250 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce408d0b91d46508252ba89d6bbe133" candidate_uuid: "cd136b80b6884be2851f12aea2d0acdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad0ba7f2f494984a3b167310151f610" is_pre_election: true
I20260926 21:41:42.192523 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:42.185259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40907 (local address 127.0.97.3:52371)
0926 21:41:42.186147 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:42.186161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:42.186313 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:42.186645 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:42.186649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:42.186671 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:42.186963 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.186971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.188035 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.188040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.190038 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:42.190053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.191655 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.191659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:42.191857 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.191896 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 21:41:42.191998 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 21:41:42.192129 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":201,"thread_start_us":361,"threads_started":1}
I20260926 21:41:42.192884 18759 raft_consensus.cc:2454] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd136b80b6884be2851f12aea2d0acdf in term 0.
I20260926 21:41:42.193624 18960 leader_election.cc:302] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd136b80b6884be2851f12aea2d0acdf, dad0ba7f2f494984a3b167310151f610; no voters:
I20260926 21:41:42.193557 18893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce408d0b91d46508252ba89d6bbe133" candidate_uuid: "cd136b80b6884be2851f12aea2d0acdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" is_pre_election: true
I20260926 21:41:42.194049 19099 raft_consensus.cc:2781] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:41:42.194056 18893 raft_consensus.cc:2454] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd136b80b6884be2851f12aea2d0acdf in term 0.
I20260926 21:41:42.194213 19099 raft_consensus.cc:482] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:41:42.194327 19099 raft_consensus.cc:3037] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:42.196609 19099 raft_consensus.cc:504] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } }
I20260926 21:41:42.197232 19099 leader_election.cc:288] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [CANDIDATE]: Term 1 election: Requested vote from peers dad0ba7f2f494984a3b167310151f610 (127.0.97.1:46065), 6a8cafdd2e1a474a8d9c58e8dcb8f489 (127.0.97.2:40907)
I20260926 21:41:42.197698 18893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce408d0b91d46508252ba89d6bbe133" candidate_uuid: "cd136b80b6884be2851f12aea2d0acdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489"
I20260926 21:41:42.197866 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce408d0b91d46508252ba89d6bbe133" candidate_uuid: "cd136b80b6884be2851f12aea2d0acdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad0ba7f2f494984a3b167310151f610"
I20260926 21:41:42.197911 18893 raft_consensus.cc:3037] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:42.198094 18759 raft_consensus.cc:3037] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:42.200103 18893 raft_consensus.cc:2454] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd136b80b6884be2851f12aea2d0acdf in term 1.
I20260926 21:41:42.200238 18759 raft_consensus.cc:2454] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd136b80b6884be2851f12aea2d0acdf in term 1.
I20260926 21:41:42.200515 18963 leader_election.cc:302] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a8cafdd2e1a474a8d9c58e8dcb8f489, cd136b80b6884be2851f12aea2d0acdf; no voters:
I20260926 21:41:42.200807 19099 raft_consensus.cc:2781] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:42.201284 19099 raft_consensus.cc:686] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 1 LEADER]: Becoming Leader. State: Replica: cd136b80b6884be2851f12aea2d0acdf, State: Running, Role: LEADER
I20260926 21:41:42.201867 19099 consensus_queue.cc:237] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } }
I20260926 21:41:42.206359 18615 catalog_manager.cc:5495] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf reported cstate change: term changed from 0 to 1, leader changed from <none> to cd136b80b6884be2851f12aea2d0acdf (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cd136b80b6884be2851f12aea2d0acdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd136b80b6884be2851f12aea2d0acdf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38901 } health_report { overall_health: HEALTHY } } }
I20260926 21:41:42.247435 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:42.241958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.0.97.3:38901)
0926 21:41:42.242173 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:42.242178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:42.242196 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:41:42.242314 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:42.242318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:42.242411 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:42.242540 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:41:42.242992 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.243704 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.245099 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.245510 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.245598 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.246224 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:41:42.246741 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:41:42.246871 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:42.247055 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 21:41:42.247126 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 21:41:42.255859 18759 raft_consensus.cc:1261] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [term 1 FOLLOWER]: Refusing update from remote peer cd136b80b6884be2851f12aea2d0acdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:42.255867 18893 raft_consensus.cc:1261] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [term 1 FOLLOWER]: Refusing update from remote peer cd136b80b6884be2851f12aea2d0acdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:42.257045 19099 consensus_queue.cc:1035] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad0ba7f2f494984a3b167310151f610" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:42.257468 19101 consensus_queue.cc:1035] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8cafdd2e1a474a8d9c58e8dcb8f489" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:42.270123 19104 mvcc.cc:205] Tried to move back new op lower bound from 7333719663629492224 to 7333719663423025152. Current Snapshot: MvccSnapshot[applied={T|T < 7333719663629492224}]
I20260926 21:41:42.286383 18615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:42.286700 18615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:42.290356 18615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46ba2cfc3e324601957b2ea2a4c92c55: Sending DeleteTablet for 3 replicas of tablet 0ce408d0b91d46508252ba89d6bbe133
I20260926 21:41:42.291625 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce408d0b91d46508252ba89d6bbe133 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56252
I20260926 21:41:42.291664 18873 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce408d0b91d46508252ba89d6bbe133 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49826
I20260926 21:41:42.292387 19007 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce408d0b91d46508252ba89d6bbe133 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44892
I20260926 21:41:42.292627 19114 tablet_replica.cc:330] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: stopping tablet replica
I20260926 21:41:42.293095 19115 tablet_replica.cc:330] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: stopping tablet replica
I20260926 21:41:42.293087 19114 raft_consensus.cc:2229] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:42.293646 19114 raft_consensus.cc:2258] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:42.293637 19115 raft_consensus.cc:2229] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:42.294260 19115 raft_consensus.cc:2258] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:42.294416 19116 tablet_replica.cc:330] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: stopping tablet replica
I20260926 21:41:42.294674 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 18676
I20260926 21:41:42.294828 19116 raft_consensus.cc:2229] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:41:42.295550 19116 raft_consensus.cc:2258] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:42.297008 19115 ts_tablet_manager.cc:1905] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:42.297819 19114 ts_tablet_manager.cc:1905] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:42.304898 19115 ts_tablet_manager.cc:1918] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:42.305225 19115 log.cc:1196] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/0ce408d0b91d46508252ba89d6bbe133
I20260926 21:41:42.305877 19115 ts_tablet_manager.cc:1939] T 0ce408d0b91d46508252ba89d6bbe133 P dad0ba7f2f494984a3b167310151f610: Deleting consensus metadata
I20260926 21:41:42.307081 19114 ts_tablet_manager.cc:1918] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:42.307271 19114 log.cc:1196] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/0ce408d0b91d46508252ba89d6bbe133
I20260926 21:41:42.307837 19114 ts_tablet_manager.cc:1939] T 0ce408d0b91d46508252ba89d6bbe133 P 6a8cafdd2e1a474a8d9c58e8dcb8f489: Deleting consensus metadata
I20260926 21:41:42.308234 18599 catalog_manager.cc:4841] TS dad0ba7f2f494984a3b167310151f610 (127.0.97.1:46065): tablet 0ce408d0b91d46508252ba89d6bbe133 (table test-table [id=b70f46d132454391bd48da6c5b071858]) successfully deleted
I20260926 21:41:42.309670 18602 catalog_manager.cc:4841] TS 6a8cafdd2e1a474a8d9c58e8dcb8f489 (127.0.97.2:40907): tablet 0ce408d0b91d46508252ba89d6bbe133 (table test-table [id=b70f46d132454391bd48da6c5b071858]) successfully deleted
I20260926 21:41:42.310766 19116 ts_tablet_manager.cc:1905] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 21:41:42 dist-test-slave-lhhq krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:41:42.318317 19116 ts_tablet_manager.cc:1918] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:42.318490 19116 log.cc:1196] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/0ce408d0b91d46508252ba89d6bbe133
I20260926 21:41:42.319041 19116 ts_tablet_manager.cc:1939] T 0ce408d0b91d46508252ba89d6bbe133 P cd136b80b6884be2851f12aea2d0acdf: Deleting consensus metadata
I20260926 21:41:42.320899 18602 catalog_manager.cc:4841] TS cd136b80b6884be2851f12aea2d0acdf (127.0.97.3:38901): tablet 0ce408d0b91d46508252ba89d6bbe133 (table test-table [id=b70f46d132454391bd48da6c5b071858]) successfully deleted
I20260926 21:41:42.323302 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:42.297422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.97.1:46065)
0926 21:41:42.297603 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:42.297609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:42.307777 (+ 10168us) server_negotiation.cc:407] Connection header received
0926 21:41:42.308301 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:42.308306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:42.308393 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:42.308474 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:42.310317 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.311133 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.312441 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.312831 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.312976 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.317037 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:42.317049 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:42.317063 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:42.317095 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:42.320192 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:42.321006 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:42.321016 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:42.321020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:42.321228 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:42.321633 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:42.321639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:42.321642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:42.322111 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:42.322427 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:42.322659 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 21:41:42.322919 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:41:42.324007 18799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:42.427148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 18676
I20260926 21:41:42.451268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 18810
Sep 26 21:41:42 dist-test-slave-lhhq krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:41:42.469157 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:42.452152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.0.97.2:40907)
0926 21:41:42.452340 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:42.452345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:42.452453 (+ 108us) server_negotiation.cc:407] Connection header received
0926 21:41:42.452633 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:42.452637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:42.452714 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:42.452806 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:42.454592 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.455379 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.457387 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.457790 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.457900 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.463217 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:42.463233 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:42.463249 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:42.463289 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:42.466320 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:42.467094 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:42.467101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:42.467104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:42.467229 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:42.467631 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:42.467635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:42.467637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:42.468025 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:42.468351 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:42.468591 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 21:41:42.468815 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:41:42.469962 18933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:42.559899 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 18810
I20260926 21:41:42.583195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 18944
Sep 26 21:41:42 dist-test-slave-lhhq krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:41:42.599855 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:42.584112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.0.97.3:38901)
0926 21:41:42.584315 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:42.584321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:42.584361 (+ 40us) server_negotiation.cc:407] Connection header received
0926 21:41:42.584474 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:42.584479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:42.584552 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:42.584673 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:42.586047 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.586875 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.588203 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:42.588668 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:42.588823 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:42.593725 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:42.593740 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:42.593756 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:42.593796 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:42.596864 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:42.597678 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:42.597685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:42.597687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:42.597846 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:42.598261 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:42.598265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:42.598267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:42.598623 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:42.598973 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:42.599266 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 21:41:42.599510 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 21:41:42.600600 19067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:42.725003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 18944
I20260926 21:41:42.744926 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 18583
I20260926 21:41:42.746111 18645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:42.857023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 18583
2026-09-26T21:41:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4470 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T21:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:41:42Z Disabled control of system clock
I20260926 21:41:42.934051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37417
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:41:43.268210 19125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:43.268518 19125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:43.268658 19125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:43.276516 19125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:43.276731 19125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:43.276824 19125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:43.276897 19125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:43.276949 19125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:43.287071 19125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:43.288393 19125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:43.290242 19125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:43.302554 19130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:43.302618 19133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:43.302662 19131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:43.303498 19125 server_base.cc:1034] running on GCE node
I20260926 21:41:43.304554 19125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:43.306068 19125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:43.307277 19125 hybrid_clock.cc:648] HybridClock initialized: now 1790458903307213 us; error 60 us; skew 500 ppm
I20260926 21:41:43.307585 19125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:43.310034 19125 webserver.cc:460] Webserver started at http://127.0.97.62:37695/ using document root <none> and password file <none>
I20260926 21:41:43.310668 19125 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:43.310752 19125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:43.311022 19125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:43.313954 19125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "2106516729e9476e99d79fec4cd13937"
format_stamp: "Formatted at 2026-09-26 21:41:43 on dist-test-slave-lhhq"
server_key: "63390b28f5e4cf8631db16b09a8ae1ce"
server_key_iv: "a7712693078fccc421a90e58156577db"
server_key_version: "encryptionkey@0"
I20260926 21:41:43.314798 19125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "2106516729e9476e99d79fec4cd13937"
format_stamp: "Formatted at 2026-09-26 21:41:43 on dist-test-slave-lhhq"
server_key: "63390b28f5e4cf8631db16b09a8ae1ce"
server_key_iv: "a7712693078fccc421a90e58156577db"
server_key_version: "encryptionkey@0"
I20260926 21:41:43.319911 19125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:41:43.323477 19139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:43.324901 19125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:41:43.325049 19125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "2106516729e9476e99d79fec4cd13937"
format_stamp: "Formatted at 2026-09-26 21:41:43 on dist-test-slave-lhhq"
server_key: "63390b28f5e4cf8631db16b09a8ae1ce"
server_key_iv: "a7712693078fccc421a90e58156577db"
server_key_version: "encryptionkey@0"
I20260926 21:41:43.325305 19125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:43.347003 19125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:43.347909 19125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:43.348161 19125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:43.366833 19125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37417
I20260926 21:41:43.366874 19190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37417 every 8 connection(s)
I20260926 21:41:43.368208 19125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:43.368530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 19125
I20260926 21:41:43.368700 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:41:43.369117 388 external_mini_cluster.cc:1428] Setting key 49132102dfcee5ac1bf13c9ab0a0cbe4
I20260926 21:41:43.373832 19191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:43.383806 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:43.371318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.0.97.62:37417)
0926 21:41:43.372360 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:43.372372 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:43.372417 (+ 45us) server_negotiation.cc:407] Connection header received
0926 21:41:43.374362 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:43.374384 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:43.374846 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:43.375169 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:43.375830 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:43.377146 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:43.379179 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:43.379958 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:43.380103 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:43.381444 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:43.381452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:43.381481 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:43.382037 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:43.382112 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:43.382133 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:43.382224 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:43.382475 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 21:41:43.382823 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":401,"threads_started":1}
I20260926 21:41:43.386483 19191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937: Bootstrap starting.
I20260926 21:41:43.389698 19191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:43.390697 19191 log.cc:824] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:43.393709 19191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937: No bootstrap required, opened a new log
I20260926 21:41:43.401398 19191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2106516729e9476e99d79fec4cd13937" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } }
I20260926 21:41:43.401860 19191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:43.401970 19191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2106516729e9476e99d79fec4cd13937, State: Initialized, Role: FOLLOWER
I20260926 21:41:43.402756 19191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2106516729e9476e99d79fec4cd13937" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } }
I20260926 21:41:43.403081 19191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:43.403224 19191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:43.403416 19191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:43.405740 19191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2106516729e9476e99d79fec4cd13937" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } }
I20260926 21:41:43.406304 19191 leader_election.cc:302] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2106516729e9476e99d79fec4cd13937; no voters:
I20260926 21:41:43.406949 19191 leader_election.cc:288] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:41:43.407145 19196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:43.407876 19196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [term 1 LEADER]: Becoming Leader. State: Replica: 2106516729e9476e99d79fec4cd13937, State: Running, Role: LEADER
I20260926 21:41:43.408444 19196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2106516729e9476e99d79fec4cd13937" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } }
I20260926 21:41:43.409232 19191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:43.411473 19198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2106516729e9476e99d79fec4cd13937. Latest consensus state: current_term: 1 leader_uuid: "2106516729e9476e99d79fec4cd13937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2106516729e9476e99d79fec4cd13937" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } } }
I20260926 21:41:43.412007 19198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:43.412483 19197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2106516729e9476e99d79fec4cd13937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2106516729e9476e99d79fec4cd13937" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } } }
I20260926 21:41:43.412861 19197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:43.414449 19204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:43.419853 19204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:43.429299 19204 catalog_manager.cc:1347] Generated new cluster ID: 2a54c2c5b8904e96a80373d3ad348909
I20260926 21:41:43.429445 19204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:43.443164 19204 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:41:43.444653 19204 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:43.457888 19204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937: Generated new TSK 0
I20260926 21:41:43.459000 19204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:41:43.474953 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37417
--builtin_ntp_servers=127.0.97.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:41:43.828593 19215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:43.828944 19215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:43.829097 19215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:43.836516 19215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:43.836678 19215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:43.836858 19215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:41:43.846310 19215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:43.847640 19215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:43.849431 19215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:43.862598 19221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:43.862665 19220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:43.862584 19223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:43.863436 19215 server_base.cc:1034] running on GCE node
I20260926 21:41:43.864058 19215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:43.865013 19215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:43.866242 19215 hybrid_clock.cc:648] HybridClock initialized: now 1790458903866178 us; error 60 us; skew 500 ppm
I20260926 21:41:43.866568 19215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:43.868759 19215 webserver.cc:460] Webserver started at http://127.0.97.1:37149/ using document root <none> and password file <none>
I20260926 21:41:43.869529 19215 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:43.869657 19215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:43.869975 19215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:43.872869 19215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "35101f2b8b8745ff9e9e8af860a95d67"
format_stamp: "Formatted at 2026-09-26 21:41:43 on dist-test-slave-lhhq"
server_key: "2abc2ab98702715ba621cdf5edec6a6c"
server_key_iv: "2b646be2f0ea33bd365a94f8051d512a"
server_key_version: "encryptionkey@0"
I20260926 21:41:43.873687 19215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "35101f2b8b8745ff9e9e8af860a95d67"
format_stamp: "Formatted at 2026-09-26 21:41:43 on dist-test-slave-lhhq"
server_key: "2abc2ab98702715ba621cdf5edec6a6c"
server_key_iv: "2b646be2f0ea33bd365a94f8051d512a"
server_key_version: "encryptionkey@0"
I20260926 21:41:43.880474 19215 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 21:41:43.884562 19229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:43.886019 19215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:41:43.886262 19215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "35101f2b8b8745ff9e9e8af860a95d67"
format_stamp: "Formatted at 2026-09-26 21:41:43 on dist-test-slave-lhhq"
server_key: "2abc2ab98702715ba621cdf5edec6a6c"
server_key_iv: "2b646be2f0ea33bd365a94f8051d512a"
server_key_version: "encryptionkey@0"
I20260926 21:41:43.886488 19215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:43.916196 19215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:43.917079 19215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:43.917387 19215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:43.918391 19215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:43.919878 19215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:43.919991 19215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:43.920125 19215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:43.920220 19215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:43.954408 19215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33009
I20260926 21:41:43.954501 19341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33009 every 8 connection(s)
I20260926 21:41:43.955770 19215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:41:43.965512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 19215
I20260926 21:41:43.965693 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:41:43.966156 388 external_mini_cluster.cc:1428] Setting key 00960093ad285b718c0be7dfc7c64046
I20260926 21:41:43.968888 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37417
--builtin_ntp_servers=127.0.97.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:41:43.969054 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:43.958621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48949 (local address 127.0.97.62:37417)
0926 21:41:43.959083 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:43.959087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:43.960205 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 21:41:43.962180 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:43.962186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:43.962290 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:43.962423 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:43.963543 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:43.964255 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:43.966095 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:43.966558 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:43.966717 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:43.968175 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:43.968179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:43.968183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:43.968200 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:43.968242 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:43.968254 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:43.968356 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:43.968595 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 21:41:43.968698 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":269,"threads_started":1}
I20260926 21:41:43.970192 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:43.959111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37417 (local address 127.0.97.1:48949)
0926 21:41:43.960033 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:43.960078 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:43.961884 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:43.962665 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:43.962673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:43.962768 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:43.963358 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:43.963379 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:43.964420 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:43.964426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:43.965927 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:43.965940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:43.966810 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:43.966818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:43.967069 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:43.967819 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:41:43.967821 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:43.967868 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:41:43.967959 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:43.968355 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:43.968370 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:43.968380 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:41:43.968584 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 21:41:43.969029 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":290,"threads_started":1}
I20260926 21:41:43.971903 19342 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37417
I20260926 21:41:43.972247 19342 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:43.973256 19342 heartbeater.cc:511] Master 127.0.97.62:37417 requested a full tablet report, sending...
I20260926 21:41:43.975612 19156 ts_manager.cc:194] Registered new tserver with Master: 35101f2b8b8745ff9e9e8af860a95d67 (127.0.97.1:33009)
I20260926 21:41:43.977670 19156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48949
W20260926 21:41:44.299723 19346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:44.300056 19346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:44.300222 19346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:44.307469 19346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:44.307605 19346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:44.307780 19346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:41:44.317093 19346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:44.318396 19346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:44.320173 19346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:44.333300 19354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:44.333369 19351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:44.333369 19352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:44.334474 19346 server_base.cc:1034] running on GCE node
I20260926 21:41:44.335081 19346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:44.336081 19346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:44.337284 19346 hybrid_clock.cc:648] HybridClock initialized: now 1790458904337238 us; error 55 us; skew 500 ppm
I20260926 21:41:44.337610 19346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:44.339815 19346 webserver.cc:460] Webserver started at http://127.0.97.2:35125/ using document root <none> and password file <none>
I20260926 21:41:44.340538 19346 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:44.340669 19346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:44.340989 19346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:44.343923 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "840658c4484640a4ae334002fcb4dade"
format_stamp: "Formatted at 2026-09-26 21:41:44 on dist-test-slave-lhhq"
server_key: "116863ddcba3ed4560ef22568291e4b0"
server_key_iv: "bbbd55011e99548a8031fffa0b04a8b5"
server_key_version: "encryptionkey@0"
I20260926 21:41:44.344707 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "840658c4484640a4ae334002fcb4dade"
format_stamp: "Formatted at 2026-09-26 21:41:44 on dist-test-slave-lhhq"
server_key: "116863ddcba3ed4560ef22568291e4b0"
server_key_iv: "bbbd55011e99548a8031fffa0b04a8b5"
server_key_version: "encryptionkey@0"
I20260926 21:41:44.349884 19346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 21:41:44.353370 19360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:44.354728 19346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:41:44.354908 19346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "840658c4484640a4ae334002fcb4dade"
format_stamp: "Formatted at 2026-09-26 21:41:44 on dist-test-slave-lhhq"
server_key: "116863ddcba3ed4560ef22568291e4b0"
server_key_iv: "bbbd55011e99548a8031fffa0b04a8b5"
server_key_version: "encryptionkey@0"
I20260926 21:41:44.355129 19346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:44.395730 19346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:44.396598 19346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:44.396878 19346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:44.397926 19346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:44.399430 19346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:44.399557 19346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:44.399682 19346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:44.399766 19346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:44.433630 19346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40067
I20260926 21:41:44.433687 19472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40067 every 8 connection(s)
I20260926 21:41:44.434981 19346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:41:44.435638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 19346
I20260926 21:41:44.435817 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:41:44.436422 388 external_mini_cluster.cc:1428] Setting key 3b4249f7e189c76f4ac5087ca8bbce9a
I20260926 21:41:44.439656 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37417
--builtin_ntp_servers=127.0.97.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:41:44.450762 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.437852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38129 (local address 127.0.97.62:37417)
0926 21:41:44.438019 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:44.438023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:44.439809 (+ 1786us) server_negotiation.cc:407] Connection header received
0926 21:41:44.442752 (+ 2943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:44.442756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:44.442825 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:44.442914 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:44.444338 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.445055 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.447659 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.448072 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.448174 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.449805 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:44.449813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:44.449821 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:44.449847 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:44.449892 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:44.449910 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:44.450005 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:44.450222 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 21:41:44.450340 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:41:44.451630 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.438475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37417 (local address 127.0.97.2:38129)
0926 21:41:44.439596 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:44.439656 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:44.442441 (+ 2785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:44.443188 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:44.443203 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:44.443340 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:44.444165 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.444192 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.445256 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.445264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.447494 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.447512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.448316 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.448325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.448671 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.449463 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:41:44.449466 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:44.449507 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:41:44.449595 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:44.450009 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:44.450017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:44.450022 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:41:44.450162 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 21:41:44.450481 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":323,"threads_started":1}
I20260926 21:41:44.453269 19473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37417
I20260926 21:41:44.453598 19473 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:44.454372 19473 heartbeater.cc:511] Master 127.0.97.62:37417 requested a full tablet report, sending...
I20260926 21:41:44.455862 19156 ts_manager.cc:194] Registered new tserver with Master: 840658c4484640a4ae334002fcb4dade (127.0.97.2:40067)
I20260926 21:41:44.456669 19156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:38129
W20260926 21:41:44.778229 19476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:44.778518 19476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:44.778609 19476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:44.785997 19476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:44.786114 19476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:44.786232 19476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:41:44.795746 19476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:44.797073 19476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:44.799011 19476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:44.812376 19481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:44.812343 19482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:44.812309 19484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:44.813278 19476 server_base.cc:1034] running on GCE node
I20260926 21:41:44.813970 19476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:44.814993 19476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:44.816262 19476 hybrid_clock.cc:648] HybridClock initialized: now 1790458904816203 us; error 37 us; skew 500 ppm
I20260926 21:41:44.816610 19476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:44.819027 19476 webserver.cc:460] Webserver started at http://127.0.97.3:44227/ using document root <none> and password file <none>
I20260926 21:41:44.819747 19476 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:44.819881 19476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:44.820207 19476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:44.823237 19476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "1c9c3f405a514ac1ab573871b26af3be"
format_stamp: "Formatted at 2026-09-26 21:41:44 on dist-test-slave-lhhq"
server_key: "deea3ea052cde0ca7f443bdb3d460110"
server_key_iv: "3455e395046bc5db8933da574dff087f"
server_key_version: "encryptionkey@0"
I20260926 21:41:44.824072 19476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1c9c3f405a514ac1ab573871b26af3be"
format_stamp: "Formatted at 2026-09-26 21:41:44 on dist-test-slave-lhhq"
server_key: "deea3ea052cde0ca7f443bdb3d460110"
server_key_iv: "3455e395046bc5db8933da574dff087f"
server_key_version: "encryptionkey@0"
I20260926 21:41:44.829483 19476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:41:44.833024 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:44.834422 19476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:41:44.834601 19476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "1c9c3f405a514ac1ab573871b26af3be"
format_stamp: "Formatted at 2026-09-26 21:41:44 on dist-test-slave-lhhq"
server_key: "deea3ea052cde0ca7f443bdb3d460110"
server_key_iv: "3455e395046bc5db8933da574dff087f"
server_key_version: "encryptionkey@0"
I20260926 21:41:44.834826 19476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:44.851600 19476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:44.852432 19476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:44.852715 19476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:44.853822 19476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:44.855262 19476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:44.855398 19476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:44.855521 19476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:44.855610 19476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:44.888810 19476 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37505
I20260926 21:41:44.888859 19602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37505 every 8 connection(s)
I20260926 21:41:44.890250 19476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:41:44.898006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 19476
I20260926 21:41:44.898192 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:41:44.898633 388 external_mini_cluster.cc:1428] Setting key f4c0148a78e7cae0556e11f1176c2b3a
I20260926 21:41:44.903285 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.892970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52075 (local address 127.0.97.62:37417)
0926 21:41:44.893223 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:44.893231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:44.894633 (+ 1402us) server_negotiation.cc:407] Connection header received
0926 21:41:44.896820 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:44.896825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:44.896921 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:44.897032 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:44.898367 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.899171 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.900710 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.901140 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.901324 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.902682 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:44.902687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:44.902690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:44.902706 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:44.902733 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:44.902744 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:44.902811 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:44.902975 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 21:41:44.903045 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 21:41:44.904227 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.893607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37417 (local address 127.0.97.3:52075)
0926 21:41:44.894475 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:44.894516 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:44.896496 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:44.897290 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:44.897298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:44.897386 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:44.898153 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.898181 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.899328 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.899333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.900557 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.900569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.901421 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.901428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.901700 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.902412 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:41:44.902415 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:44.902453 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:41:44.902525 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:44.902810 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:44.902816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:44.902821 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 21:41:44.902948 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 21:41:44.903222 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":278,"threads_started":1}
I20260926 21:41:44.905483 19603 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37417
I20260926 21:41:44.905733 19603 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:44.906461 19603 heartbeater.cc:511] Master 127.0.97.62:37417 requested a full tablet report, sending...
I20260926 21:41:44.907764 19156 ts_manager.cc:194] Registered new tserver with Master: 1c9c3f405a514ac1ab573871b26af3be (127.0.97.3:37505)
I20260926 21:41:44.908515 19156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52075
I20260926 21:41:44.913542 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:41:44.923213 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.917723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.0.97.62:37417)
0926 21:41:44.917874 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:44.917878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:44.918358 (+ 480us) server_negotiation.cc:407] Connection header received
0926 21:41:44.918489 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:44.918493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:44.918550 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:44.918631 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:44.919085 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.919742 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.921020 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.921507 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.921597 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.922616 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:44.922621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:44.922624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:44.922641 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:44.922672 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:44.922683 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:44.922791 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:44.922943 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 21:41:44.922986 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 21:41:44.926940 19156 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:41:44.934082 19156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:41:44.967756 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.946537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.0.1:47504)
0926 21:41:44.949059 (+ 2522us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:44.949103 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:44.949443 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:44.952623 (+ 3180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:44.952631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:44.952671 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:44.953264 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.953281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.957865 (+ 4584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.957873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.962827 (+ 4954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.962851 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.966543 (+ 3692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.966549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.966851 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.966907 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 21:41:44.967051 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 21:41:44.967280 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2240,"mutex_wait_us":31,"spinlock_wait_cycles":30976,"thread_start_us":2063,"threads_started":1}
I20260926 21:41:44.968158 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.947114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.0.97.1:33009)
0926 21:41:44.948600 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:44.948612 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:44.949254 (+ 642us) server_negotiation.cc:407] Connection header received
0926 21:41:44.951247 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:44.951261 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:44.951544 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:44.951793 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:44.953520 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.957610 (+ 4090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.963137 (+ 5527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.966336 (+ 3199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.966484 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.966781 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:44.967618 (+ 837us) server_negotiation.cc:299] Negotiation successful
0926 21:41:44.967802 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":1174,"threads_started":1}
I20260926 21:41:44.970507 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.946595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40067 (local address 127.0.0.1:50412)
0926 21:41:44.949106 (+ 2511us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:44.949129 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:44.949422 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:44.950666 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:44.950677 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:44.950719 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:44.951277 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.951294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.965326 (+ 14032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.965335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.967587 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.967603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.969550 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.969557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.969751 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.969868 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 21:41:44.969995 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 21:41:44.970133 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2366,"mutex_wait_us":100,"spinlock_wait_cycles":2304,"thread_start_us":2205,"threads_started":1}
I20260926 21:41:44.970553 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.947161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.0.97.2:40067)
0926 21:41:44.948393 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:44.948408 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:44.949251 (+ 843us) server_negotiation.cc:407] Connection header received
0926 21:41:44.949691 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:44.949705 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:44.950036 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:44.950261 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:44.951547 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.965068 (+ 13521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.967840 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.969331 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.969484 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.969726 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:44.970042 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 21:41:44.970208 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":807,"threads_started":1}
I20260926 21:41:44.972231 19277 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d (DEFAULT_TABLE table=test-table [id=bd79633000644381933afee81a05ecb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:44.973855 19277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:44.974259 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.949532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37505 (local address 127.0.0.1:44728)
0926 21:41:44.966049 (+ 16517us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:44.966073 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:44.966208 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:44.966868 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:44.966873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:44.966898 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:44.967343 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.967355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.969358 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.969363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.972035 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:44.972048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.973434 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.973441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:44.973606 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.973649 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 21:41:44.973749 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 21:41:44.973873 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16381,"thread_start_us":16224,"threads_started":1}
I20260926 21:41:44.974304 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:44.949899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.0.97.3:37505)
0926 21:41:44.965504 (+ 15605us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:44.965514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:44.966203 (+ 689us) server_negotiation.cc:407] Connection header received
0926 21:41:44.966352 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:44.966360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:44.966633 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:44.966832 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:44.967566 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.969136 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.972219 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:44.973229 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:44.973391 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:44.973665 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:44.973786 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 21:41:44.973948 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15206,"thread_start_us":15287,"threads_started":1}
I20260926 21:41:44.974519 19408 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d (DEFAULT_TABLE table=test-table [id=bd79633000644381933afee81a05ecb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:44.975903 19408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:44.977558 19538 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d (DEFAULT_TABLE table=test-table [id=bd79633000644381933afee81a05ecb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:44.979012 19538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:44.980945 19342 heartbeater.cc:503] Master 127.0.97.62:37417 was elected leader, sending a full tablet report...
I20260926 21:41:44.989319 19621 tablet_bootstrap.cc:492] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67: Bootstrap starting.
I20260926 21:41:44.991812 19623 tablet_bootstrap.cc:492] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: Bootstrap starting.
I20260926 21:41:44.993011 19622 tablet_bootstrap.cc:492] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: Bootstrap starting.
I20260926 21:41:44.993814 19621 tablet_bootstrap.cc:654] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:44.994817 19621 log.cc:824] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:44.995508 19623 tablet_bootstrap.cc:654] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:44.996385 19622 tablet_bootstrap.cc:654] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:44.996704 19623 log.cc:824] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:44.997637 19622 log.cc:824] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:44.997937 19621 tablet_bootstrap.cc:492] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67: No bootstrap required, opened a new log
I20260926 21:41:44.998189 19621 ts_tablet_manager.cc:1397] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 21:41:45.000849 19622 tablet_bootstrap.cc:492] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: No bootstrap required, opened a new log
I20260926 21:41:45.001113 19622 ts_tablet_manager.cc:1397] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 21:41:45.002931 19623 tablet_bootstrap.cc:492] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: No bootstrap required, opened a new log
I20260926 21:41:45.003561 19623 ts_tablet_manager.cc:1397] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260926 21:41:45.005690 19621 raft_consensus.cc:348] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
I20260926 21:41:45.006270 19621 raft_consensus.cc:374] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:45.006421 19621 raft_consensus.cc:729] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35101f2b8b8745ff9e9e8af860a95d67, State: Initialized, Role: FOLLOWER
I20260926 21:41:45.007052 19621 consensus_queue.cc:260] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
I20260926 21:41:45.009953 19621 ts_tablet_manager.cc:1428] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260926 21:41:45.012251 19622 raft_consensus.cc:348] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
I20260926 21:41:45.012864 19622 raft_consensus.cc:374] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:45.012986 19622 raft_consensus.cc:729] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9c3f405a514ac1ab573871b26af3be, State: Initialized, Role: FOLLOWER
I20260926 21:41:45.013721 19622 consensus_queue.cc:260] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
I20260926 21:41:45.014082 19623 raft_consensus.cc:348] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
I20260926 21:41:45.014603 19623 raft_consensus.cc:374] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:45.014761 19623 raft_consensus.cc:729] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840658c4484640a4ae334002fcb4dade, State: Initialized, Role: FOLLOWER
I20260926 21:41:45.015158 19603 heartbeater.cc:503] Master 127.0.97.62:37417 was elected leader, sending a full tablet report...
I20260926 21:41:45.015568 19622 ts_tablet_manager.cc:1428] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 21:41:45.015601 19623 consensus_queue.cc:260] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
I20260926 21:41:45.016800 19473 heartbeater.cc:503] Master 127.0.97.62:37417 was elected leader, sending a full tablet report...
I20260926 21:41:45.017477 19623 ts_tablet_manager.cc:1428] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260926 21:41:45.114281 19628 raft_consensus.cc:482] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:41:45.114660 19628 raft_consensus.cc:504] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
I20260926 21:41:45.116559 19628 leader_election.cc:288] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35101f2b8b8745ff9e9e8af860a95d67 (127.0.97.1:33009), 840658c4484640a4ae334002fcb4dade (127.0.97.2:40067)
I20260926 21:41:45.122042 19629 raft_consensus.cc:482] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:41:45.122414 19629 raft_consensus.cc:504] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
I20260926 21:41:45.123769 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.117042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40067 (local address 127.0.97.3:34221)
0926 21:41:45.117281 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.117302 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:45.117490 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:45.117927 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:45.117940 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:45.117983 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:45.118220 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:45.118230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.119592 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.119598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:45.121313 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:45.121325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.123058 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.123062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:45.123179 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.123201 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:41:45.123313 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 21:41:45.123452 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 21:41:45.123759 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.117041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34221 (local address 127.0.97.2:40067)
0926 21:41:45.117261 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:45.117268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:45.117411 (+ 143us) server_negotiation.cc:407] Connection header received
0926 21:41:45.117636 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:45.117640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:45.117706 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:45.117858 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:45.118413 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.119461 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.121470 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.122797 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.122914 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.122987 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:45.123364 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 21:41:45.123484 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:41:45.124959 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.117138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52781 (local address 127.0.97.1:33009)
0926 21:41:45.117350 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:45.117355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:45.118466 (+ 1111us) server_negotiation.cc:407] Connection header received
0926 21:41:45.118625 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:45.118629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:45.118697 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:45.118765 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:45.119312 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.120322 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.122642 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.123786 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.123939 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.124047 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:45.124570 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 21:41:45.124717 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:41:45.125048 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.117544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.97.3:52781)
0926 21:41:45.118330 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.118349 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:45.118483 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:45.118812 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:45.118817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:45.118843 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:45.119156 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:45.119166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.120526 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.120530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:45.122439 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:45.122456 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.124148 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.124153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:45.124341 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.124373 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 21:41:45.124511 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 21:41:45.124647 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":3,"thread_start_us":416,"threads_started":1}
I20260926 21:41:45.125223 19428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0d61a4cbc5467e9a6e88a414c3b71d" candidate_uuid: "1c9c3f405a514ac1ab573871b26af3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840658c4484640a4ae334002fcb4dade" is_pre_election: true
I20260926 21:41:45.125746 19629 leader_election.cc:288] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35101f2b8b8745ff9e9e8af860a95d67 (127.0.97.1:33009), 1c9c3f405a514ac1ab573871b26af3be (127.0.97.3:37505)
I20260926 21:41:45.125800 19428 raft_consensus.cc:2454] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9c3f405a514ac1ab573871b26af3be in term 0.
I20260926 21:41:45.126443 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0d61a4cbc5467e9a6e88a414c3b71d" candidate_uuid: "1c9c3f405a514ac1ab573871b26af3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35101f2b8b8745ff9e9e8af860a95d67" is_pre_election: true
I20260926 21:41:45.127038 19297 raft_consensus.cc:2454] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9c3f405a514ac1ab573871b26af3be in term 0.
I20260926 21:41:45.127637 19493 leader_election.cc:302] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9c3f405a514ac1ab573871b26af3be, 840658c4484640a4ae334002fcb4dade; no voters:
I20260926 21:41:45.128041 19628 raft_consensus.cc:2781] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:41:45.128237 19628 raft_consensus.cc:482] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:41:45.128351 19628 raft_consensus.cc:3037] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:45.131796 19628 raft_consensus.cc:504] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
I20260926 21:41:45.132532 19628 leader_election.cc:288] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [CANDIDATE]: Term 1 election: Requested vote from peers 35101f2b8b8745ff9e9e8af860a95d67 (127.0.97.1:33009), 840658c4484640a4ae334002fcb4dade (127.0.97.2:40067)
I20260926 21:41:45.133073 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0d61a4cbc5467e9a6e88a414c3b71d" candidate_uuid: "1c9c3f405a514ac1ab573871b26af3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35101f2b8b8745ff9e9e8af860a95d67"
I20260926 21:41:45.133081 19428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0d61a4cbc5467e9a6e88a414c3b71d" candidate_uuid: "1c9c3f405a514ac1ab573871b26af3be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840658c4484640a4ae334002fcb4dade"
I20260926 21:41:45.133303 19297 raft_consensus.cc:3037] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:45.133360 19428 raft_consensus.cc:3037] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:45.136168 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.125066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.97.2:57575)
0926 21:41:45.127111 (+ 2045us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.127132 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:45.127263 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:45.127918 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:45.127923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:45.127951 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:45.128749 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:45.128764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.130896 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.130903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:45.133266 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:45.133282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.135375 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.135381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:45.135554 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.135589 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 21:41:45.135694 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 21:41:45.135836 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1901,"thread_start_us":1660,"threads_started":1}
I20260926 21:41:45.136385 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.126074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43149 (local address 127.0.97.3:37505)
0926 21:41:45.126262 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:45.126270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:45.129572 (+ 3302us) server_negotiation.cc:407] Connection header received
0926 21:41:45.129751 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:45.129757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:45.129833 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:45.129931 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:45.130483 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.131575 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.134108 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.135293 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.135387 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.135487 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:45.136027 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 21:41:45.136155 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:41:45.136881 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.125710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57575 (local address 127.0.97.1:33009)
0926 21:41:45.127532 (+ 1822us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:45.127541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:45.127570 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:41:45.127643 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:45.127651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:45.127749 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:45.127878 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:45.129518 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.130734 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.133524 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.134967 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.135092 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.135237 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:45.136364 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0926 21:41:45.136518 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1656,"thread_start_us":2389,"threads_started":1}
I20260926 21:41:45.137027 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.126088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37505 (local address 127.0.97.2:43149)
0926 21:41:45.129468 (+ 3380us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.129490 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:45.129617 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:45.129974 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:45.129980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:45.130006 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:45.130324 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:45.130335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.131749 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.131755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:45.133883 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:45.133900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.135661 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.135670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:45.135848 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.135881 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:41:45.136565 (+ 684us) client_negotiation.cc:240] Negotiation successful
0926 21:41:45.136693 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3231,"thread_start_us":3041,"threads_started":1}
I20260926 21:41:45.137097 19428 raft_consensus.cc:2454] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9c3f405a514ac1ab573871b26af3be in term 1.
I20260926 21:41:45.137449 19297 raft_consensus.cc:2454] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9c3f405a514ac1ab573871b26af3be in term 1.
I20260926 21:41:45.137547 19296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0d61a4cbc5467e9a6e88a414c3b71d" candidate_uuid: "840658c4484640a4ae334002fcb4dade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35101f2b8b8745ff9e9e8af860a95d67" is_pre_election: true
I20260926 21:41:45.137964 19296 raft_consensus.cc:2379] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 840658c4484640a4ae334002fcb4dade in current term 1: Already voted for candidate 1c9c3f405a514ac1ab573871b26af3be in this term.
I20260926 21:41:45.137907 19558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0d61a4cbc5467e9a6e88a414c3b71d" candidate_uuid: "840658c4484640a4ae334002fcb4dade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9c3f405a514ac1ab573871b26af3be" is_pre_election: true
I20260926 21:41:45.137913 19493 leader_election.cc:302] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9c3f405a514ac1ab573871b26af3be, 840658c4484640a4ae334002fcb4dade; no voters:
I20260926 21:41:45.138491 19558 raft_consensus.cc:2379] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 840658c4484640a4ae334002fcb4dade in current term 1: Already voted for candidate 1c9c3f405a514ac1ab573871b26af3be in this term.
I20260926 21:41:45.138732 19628 raft_consensus.cc:2781] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:45.139236 19628 raft_consensus.cc:686] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 1 LEADER]: Becoming Leader. State: Replica: 1c9c3f405a514ac1ab573871b26af3be, State: Running, Role: LEADER
I20260926 21:41:45.139354 19363 leader_election.cc:302] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 840658c4484640a4ae334002fcb4dade; no voters: 1c9c3f405a514ac1ab573871b26af3be, 35101f2b8b8745ff9e9e8af860a95d67
I20260926 21:41:45.140106 19629 raft_consensus.cc:2726] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 21:41:45.140074 19628 consensus_queue.cc:237] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } }
W20260926 21:41:45.141983 19604 tablet.cc:2369] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:45.144763 19156 catalog_manager.cc:5495] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c9c3f405a514ac1ab573871b26af3be (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1c9c3f405a514ac1ab573871b26af3be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c9c3f405a514ac1ab573871b26af3be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37505 } health_report { overall_health: HEALTHY } } }
W20260926 21:41:45.187286 19474 tablet.cc:2369] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:45.198436 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.193259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.0.97.3:37505)
0926 21:41:45.193464 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:45.193469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:45.193487 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:41:45.193590 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:45.193594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:45.193700 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:45.193818 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:41:45.194166 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.194913 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.196252 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.196653 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.196741 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.197260 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:41:45.197770 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:41:45.197899 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:45.198042 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 21:41:45.198105 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:41:45.207090 19296 raft_consensus.cc:1261] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 1 FOLLOWER]: Refusing update from remote peer 1c9c3f405a514ac1ab573871b26af3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:45.207288 19428 raft_consensus.cc:1261] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 1 FOLLOWER]: Refusing update from remote peer 1c9c3f405a514ac1ab573871b26af3be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:45.207983 19634 consensus_queue.cc:1035] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "35101f2b8b8745ff9e9e8af860a95d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 21:41:45.208357 19343 tablet.cc:2369] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:45.208436 19628 consensus_queue.cc:1035] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [LEADER]: Connected to new peer: Peer: permanent_uuid: "840658c4484640a4ae334002fcb4dade" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:45.245004 19155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:45.245370 19155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:45.249382 19155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2106516729e9476e99d79fec4cd13937: Sending DeleteTablet for 3 replicas of tablet 7c0d61a4cbc5467e9a6e88a414c3b71d
I20260926 21:41:45.250557 19277 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:45 UTC) from {username='slave'} at 127.0.0.1:47504
I20260926 21:41:45.250715 19408 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:45 UTC) from {username='slave'} at 127.0.0.1:50412
I20260926 21:41:45.251816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 19215
I20260926 21:41:45.258733 19538 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:45 UTC) from {username='slave'} at 127.0.0.1:44728
I20260926 21:41:45.259763 19649 tablet_replica.cc:330] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: stopping tablet replica
I20260926 21:41:45.260344 19649 raft_consensus.cc:2229] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:45.260960 19649 raft_consensus.cc:2258] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:45.264056 19649 ts_tablet_manager.cc:1905] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:45.268162 19651 tablet_replica.cc:330] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: stopping tablet replica
I20260926 21:41:45.268639 19651 raft_consensus.cc:2229] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:41:45.269357 19651 raft_consensus.cc:2258] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:45.271538 19648 tablet_replica.cc:330] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67: stopping tablet replica
I20260926 21:41:45.271977 19648 raft_consensus.cc:2229] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:45.272200 19651 ts_tablet_manager.cc:1905] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:45.272477 19648 raft_consensus.cc:2258] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:45.274554 19648 ts_tablet_manager.cc:1905] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 35101f2b8b8745ff9e9e8af860a95d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:45.274756 19649 ts_tablet_manager.cc:1918] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:45.274972 19649 log.cc:1196] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/7c0d61a4cbc5467e9a6e88a414c3b71d
I20260926 21:41:45.275602 19649 ts_tablet_manager.cc:1939] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 840658c4484640a4ae334002fcb4dade: Deleting consensus metadata
I20260926 21:41:45.277689 19142 catalog_manager.cc:4841] TS 840658c4484640a4ae334002fcb4dade (127.0.97.2:40067): tablet 7c0d61a4cbc5467e9a6e88a414c3b71d (table test-table [id=bd79633000644381933afee81a05ecb1]) successfully deleted
I20260926 21:41:45.278071 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.253452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.0.97.1:33009)
0926 21:41:45.253643 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:45.253649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:45.264457 (+ 10808us) server_negotiation.cc:407] Connection header received
0926 21:41:45.264953 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:45.264959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:45.265066 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:45.265140 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:45.265772 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.266857 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.268794 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.269767 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.269921 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.276258 (+ 6337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:45.276269 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:45.276285 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:45.277061 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:45.277132 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:45.277211 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:45.277304 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:45.277578 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 21:41:45.277720 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:41:45.279271 19337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:45.280474 19651 ts_tablet_manager.cc:1918] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:45.280689 19651 log.cc:1196] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/7c0d61a4cbc5467e9a6e88a414c3b71d
I20260926 21:41:45.281388 19651 ts_tablet_manager.cc:1939] T 7c0d61a4cbc5467e9a6e88a414c3b71d P 1c9c3f405a514ac1ab573871b26af3be: Deleting consensus metadata
I20260926 21:41:45.283704 19142 catalog_manager.cc:4841] TS 1c9c3f405a514ac1ab573871b26af3be (127.0.97.3:37505): tablet 7c0d61a4cbc5467e9a6e88a414c3b71d (table test-table [id=bd79633000644381933afee81a05ecb1]) successfully deleted
I20260926 21:41:45.373174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 19215
W20260926 21:41:45.399044 19143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33009 (error 108)
I20260926 21:41:45.399803 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 19346
I20260926 21:41:45.401145 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.400581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.0.1:47520)
0926 21:41:45.400856 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.400978 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260926 21:41:45.401702 19143 catalog_manager.cc:4584] TS 35101f2b8b8745ff9e9e8af860a95d67 (127.0.97.1:33009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c0d61a4cbc5467e9a6e88a414c3b71d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33009: connect: Connection refused (error 111)
I20260926 21:41:45.408108 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.400654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.0.97.2:40067)
0926 21:41:45.400835 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:45.400842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:45.400949 (+ 107us) server_negotiation.cc:407] Connection header received
0926 21:41:45.401151 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:45.401163 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:45.401328 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:45.401447 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:45.401891 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.402623 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.404164 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.404824 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.404962 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.406120 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:45.406133 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:45.406152 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:45.406922 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:45.407010 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:45.407035 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:45.407146 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:45.407652 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 21:41:45.407770 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 21:41:45.408872 19468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:45.455668 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.455059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.0.1:47528)
0926 21:41:45.455336 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.455462 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 21:41:45.457021 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.456572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.0.1:47530)
0926 21:41:45.456753 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.456853 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 21:41:45.500749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 19346
I20260926 21:41:45.517814 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.517243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.0.1:47544)
0926 21:41:45.517489 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.517605 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 21:41:45.519012 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.518583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.0.1:47546)
0926 21:41:45.518756 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.518862 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 21:41:45.519687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 19476
I20260926 21:41:45.527151 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.520446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.0.97.3:37505)
0926 21:41:45.520636 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:45.520641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:45.520660 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:41:45.520854 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:45.520858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:45.520938 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:45.521058 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:45.521505 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.522289 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.523564 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:45.524017 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:45.524157 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:45.525385 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:45.525397 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:45.525416 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:45.526287 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:45.526355 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:45.526382 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:45.526512 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:45.526661 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 21:41:45.526771 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:41:45.527915 19598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:45.600005 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.599496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.0.1:47554)
0926 21:41:45.599724 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.599817 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 21:41:45.601164 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.600777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.0.1:47558)
0926 21:41:45.600979 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.601065 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 21:41:45.648990 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 19476
I20260926 21:41:45.668473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 19125
I20260926 21:41:45.669580 19186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:45.787580 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:45.786758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33009 (local address 127.0.0.1:47562)
0926 21:41:45.787066 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:45.787192 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 21:41:45.787701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 19125
2026-09-26T21:41:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2925 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:41:45 dist-test-slave-lhhq krb5kdc[19659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:41:45 dist-test-slave-lhhq krb5kdc[19659](info): set up 2 sockets
Sep 26 21:41:45 dist-test-slave-lhhq krb5kdc[19659](info): commencing operation
krb5kdc: starting...
W20260926 21:41:46.897840 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:41:47 dist-test-slave-lhhq krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:41:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:41:47.149926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39365
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:47.479017 19672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:47.479408 19672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:47.479609 19672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:47.487149 19672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:47.487308 19672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:47.487394 19672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:47.487504 19672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:47.487571 19672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:47.497874 19672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:47.499229 19672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:47.500537 19672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:47.512657 19677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:47.512703 19678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:47.512622 19680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:47.513708 19672 server_base.cc:1034] running on GCE node
I20260926 21:41:47.514510 19672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:47.515892 19672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:47.517150 19672 hybrid_clock.cc:648] HybridClock initialized: now 1790458907517092 us; error 37 us; skew 500 ppm
I20260926 21:41:47.517710 19672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:47 dist-test-slave-lhhq krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:47.521163 19672 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:41:47.523140 19672 webserver.cc:460] Webserver started at http://127.0.97.62:37359/ using document root <none> and password file <none>
I20260926 21:41:47.523964 19672 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:47.524137 19672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:47.524565 19672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:47.528177 19672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "6a628efc2b2d4c40aba7296b5467b17b"
format_stamp: "Formatted at 2026-09-26 21:41:47 on dist-test-slave-lhhq"
server_key: "0cb33fceac6bb30b88445096104dfe67"
server_key_iv: "54ee6b8615dec5b49a1eb61f16e119d8"
server_key_version: "encryptionkey@0"
I20260926 21:41:47.529095 19672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "6a628efc2b2d4c40aba7296b5467b17b"
format_stamp: "Formatted at 2026-09-26 21:41:47 on dist-test-slave-lhhq"
server_key: "0cb33fceac6bb30b88445096104dfe67"
server_key_iv: "54ee6b8615dec5b49a1eb61f16e119d8"
server_key_version: "encryptionkey@0"
I20260926 21:41:47.535300 19672 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260926 21:41:47.538946 19687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:47.540377 19672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:41:47.540577 19672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "6a628efc2b2d4c40aba7296b5467b17b"
format_stamp: "Formatted at 2026-09-26 21:41:47 on dist-test-slave-lhhq"
server_key: "0cb33fceac6bb30b88445096104dfe67"
server_key_iv: "54ee6b8615dec5b49a1eb61f16e119d8"
server_key_version: "encryptionkey@0"
I20260926 21:41:47.540798 19672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:47.555604 19672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:47.559613 19672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:47.559924 19672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:47.578413 19672 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39365
I20260926 21:41:47.578444 19738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39365 every 8 connection(s)
I20260926 21:41:47.579847 19672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:47.583766 19739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:47.584116 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 19672
I20260926 21:41:47.584327 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:41:47.584810 388 external_mini_cluster.cc:1428] Setting key 269915e486419921a26e7abc3a67d44d
I20260926 21:41:47.597110 19739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b: Bootstrap starting.
Sep 26 21:41:47 dist-test-slave-lhhq krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:47.600521 19739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:47.601653 19739 log.cc:824] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:47.604696 19739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b: No bootstrap required, opened a new log
I20260926 21:41:47.609045 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:47.586745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.0.97.62:39365)
0926 21:41:47.587790 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:47.587814 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:47.588132 (+ 318us) server_negotiation.cc:407] Connection header received
0926 21:41:47.590687 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:47.590714 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:47.591140 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:47.591557 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:47.593004 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:47.594376 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:47.595951 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:47.596616 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:47.596837 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:47.601099 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:47.601107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:47.601122 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:47.601158 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:47.604479 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:47.605346 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:47.605354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:47.605368 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:47.605562 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:47.605970 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:47.605975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:47.605978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:47.606567 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:47.606925 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:47.607384 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 21:41:47.607879 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":487,"threads_started":1}
I20260926 21:41:47.613004 19739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a628efc2b2d4c40aba7296b5467b17b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39365 } }
I20260926 21:41:47.613468 19739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:47.613602 19739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a628efc2b2d4c40aba7296b5467b17b, State: Initialized, Role: FOLLOWER
I20260926 21:41:47.614158 19739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a628efc2b2d4c40aba7296b5467b17b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39365 } }
I20260926 21:41:47.614363 19739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:47.614521 19739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:47.614717 19739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:47.617452 19739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a628efc2b2d4c40aba7296b5467b17b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39365 } }
I20260926 21:41:47.617964 19739 leader_election.cc:302] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a628efc2b2d4c40aba7296b5467b17b; no voters:
I20260926 21:41:47.618532 19739 leader_election.cc:288] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:41:47.618678 19744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:47.619360 19744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [term 1 LEADER]: Becoming Leader. State: Replica: 6a628efc2b2d4c40aba7296b5467b17b, State: Running, Role: LEADER
I20260926 21:41:47.619863 19744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a628efc2b2d4c40aba7296b5467b17b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39365 } }
I20260926 21:41:47.620602 19739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:47.623817 19746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a628efc2b2d4c40aba7296b5467b17b. Latest consensus state: current_term: 1 leader_uuid: "6a628efc2b2d4c40aba7296b5467b17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a628efc2b2d4c40aba7296b5467b17b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39365 } } }
I20260926 21:41:47.624840 19746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:47.624135 19745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a628efc2b2d4c40aba7296b5467b17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a628efc2b2d4c40aba7296b5467b17b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39365 } } }
I20260926 21:41:47.625378 19745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:47.628847 19753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:47.635485 19753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:47.644742 19753 catalog_manager.cc:1347] Generated new cluster ID: 77012682fb024cd2b85dcde8917be11c
I20260926 21:41:47.644879 19753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:47.668993 19753 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:41:47.670743 19753 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:47.680886 19753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b: Generated new TSK 0
I20260926 21:41:47.682077 19753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 21:41:47.757447 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39365
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:48.099231 19765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:48.099529 19765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:48.099706 19765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:48.107275 19765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:48.107400 19765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:48.107612 19765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:41:48.117995 19765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:48.119401 19765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:48.120780 19765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:48.135018 19770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:48.135223 19773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:48.135341 19771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:48.136031 19765 server_base.cc:1034] running on GCE node
I20260926 21:41:48.136621 19765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:48.137609 19765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:48.138852 19765 hybrid_clock.cc:648] HybridClock initialized: now 1790458908138803 us; error 61 us; skew 500 ppm
I20260926 21:41:48.139197 19765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:48 dist-test-slave-lhhq krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:48.142500 19765 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 21:41:48.143960 19765 webserver.cc:460] Webserver started at http://127.0.97.1:45593/ using document root <none> and password file <none>
I20260926 21:41:48.144742 19765 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:48.144847 19765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:48.145262 19765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:48.148276 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "da74f1876def430cb2a337a0cda654f3"
format_stamp: "Formatted at 2026-09-26 21:41:48 on dist-test-slave-lhhq"
server_key: "96336eb89e2d64c67ba3ddddbcafab35"
server_key_iv: "9958c1bfc5efd3bf8e6a19695de72e4c"
server_key_version: "encryptionkey@0"
I20260926 21:41:48.149039 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "da74f1876def430cb2a337a0cda654f3"
format_stamp: "Formatted at 2026-09-26 21:41:48 on dist-test-slave-lhhq"
server_key: "96336eb89e2d64c67ba3ddddbcafab35"
server_key_iv: "9958c1bfc5efd3bf8e6a19695de72e4c"
server_key_version: "encryptionkey@0"
I20260926 21:41:48.154273 19765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 21:41:48.158094 19780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:48.159602 19765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:41:48.159776 19765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "da74f1876def430cb2a337a0cda654f3"
format_stamp: "Formatted at 2026-09-26 21:41:48 on dist-test-slave-lhhq"
server_key: "96336eb89e2d64c67ba3ddddbcafab35"
server_key_iv: "9958c1bfc5efd3bf8e6a19695de72e4c"
server_key_version: "encryptionkey@0"
I20260926 21:41:48.159997 19765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:48.181737 19765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:48.186107 19765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:48.186436 19765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:48.187450 19765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:48.188839 19765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:48.188918 19765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:48.189060 19765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:48.189119 19765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:48.223295 19765 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35971
I20260926 21:41:48.223393 19892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35971 every 8 connection(s)
I20260926 21:41:48.224624 19765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:41:48.226739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 19765
I20260926 21:41:48.226922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:41:48.227358 388 external_mini_cluster.cc:1428] Setting key bc194492b4074eec5189f7f79685811f
Sep 26 21:41:48 dist-test-slave-lhhq krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:48.245767 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:48.227607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47759 (local address 127.0.97.62:39365)
0926 21:41:48.228110 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:48.228118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:48.228872 (+ 754us) server_negotiation.cc:407] Connection header received
0926 21:41:48.231138 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:48.231143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:48.231238 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:48.231356 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:48.232913 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:48.233746 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:48.235268 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:48.235680 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:48.235832 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:48.239873 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:48.239878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:48.239881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:48.239915 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:48.242585 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:48.243325 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:48.243332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:48.243335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:48.243459 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:48.243888 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:48.243894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:48.243897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:48.244202 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:48.244361 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:48.245107 (+ 746us) server_negotiation.cc:299] Negotiation successful
0926 21:41:48.245364 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260926 21:41:48.246785 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:48.227897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39365 (local address 127.0.97.1:47759)
0926 21:41:48.228723 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:48.228762 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:48.230880 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:48.231611 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:48.231619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:48.232176 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:48.232739 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:48.232758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:48.233896 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:48.233908 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:48.235123 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:48.235135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:48.235918 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:48.235927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:48.236225 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:48.237026 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:41:48.237062 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:48.239621 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:48.242736 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:48.242745 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:48.242762 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:48.243189 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:48.243571 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:48.243576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:48.243580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:48.243774 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:48.244334 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:48.244342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:48.244721 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 21:41:48.245023 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 21:41:48.245611 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":372,"threads_started":1}
I20260926 21:41:48.248832 19893 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39365
I20260926 21:41:48.249223 19893 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:48.250296 19893 heartbeater.cc:511] Master 127.0.97.62:39365 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 21:41:48.253364 19704 ts_manager.cc:194] Registered new tserver with Master: da74f1876def430cb2a337a0cda654f3 (127.0.97.1:35971)
I20260926 21:41:48.254915 19704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 21:41:48.285024 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39365
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:48.635185 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:48.635484 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:48.635581 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:48.642796 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:48.642922 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:48.643038 19899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:41:48.653409 19899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:48.654731 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:48.656041 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:48.669589 19905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:48.669682 19907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:48.670111 19904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:48.670794 19899 server_base.cc:1034] running on GCE node
I20260926 21:41:48.671445 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:48.672405 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:48.673606 19899 hybrid_clock.cc:648] HybridClock initialized: now 1790458908673553 us; error 39 us; skew 500 ppm
I20260926 21:41:48.673942 19899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:48 dist-test-slave-lhhq krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:48.677169 19899 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 21:41:48.678663 19899 webserver.cc:460] Webserver started at http://127.0.97.2:40001/ using document root <none> and password file <none>
I20260926 21:41:48.679446 19899 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:48.679574 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:48.679895 19899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:48.682847 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "ff748379baee43c2b1348fa20f26d9d7"
format_stamp: "Formatted at 2026-09-26 21:41:48 on dist-test-slave-lhhq"
server_key: "d7d6bfbdcd82831a9f5b9458d81886f2"
server_key_iv: "91d27deff28676df25f9e85035ea1f78"
server_key_version: "encryptionkey@0"
I20260926 21:41:48.683882 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ff748379baee43c2b1348fa20f26d9d7"
format_stamp: "Formatted at 2026-09-26 21:41:48 on dist-test-slave-lhhq"
server_key: "d7d6bfbdcd82831a9f5b9458d81886f2"
server_key_iv: "91d27deff28676df25f9e85035ea1f78"
server_key_version: "encryptionkey@0"
I20260926 21:41:48.688971 19899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 21:41:48.692435 19914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:48.693874 19899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:41:48.694049 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "ff748379baee43c2b1348fa20f26d9d7"
format_stamp: "Formatted at 2026-09-26 21:41:48 on dist-test-slave-lhhq"
server_key: "d7d6bfbdcd82831a9f5b9458d81886f2"
server_key_iv: "91d27deff28676df25f9e85035ea1f78"
server_key_version: "encryptionkey@0"
I20260926 21:41:48.694255 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:48.719951 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:48.724838 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:48.725123 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:48.726151 19899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:48.727517 19899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:48.727592 19899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:48.727761 19899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:48.727813 19899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:48.760869 19899 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34257
I20260926 21:41:48.760915 20026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34257 every 8 connection(s)
I20260926 21:41:48.762219 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:41:48.770938 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 19899
I20260926 21:41:48.771108 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:41:48.771559 388 external_mini_cluster.cc:1428] Setting key fdfc9597e7a8a930b571be72f232acd8
Sep 26 21:41:48 dist-test-slave-lhhq krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:48.782625 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:48.764846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52823 (local address 127.0.97.62:39365)
0926 21:41:48.765305 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:48.765311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:48.766347 (+ 1036us) server_negotiation.cc:407] Connection header received
0926 21:41:48.768101 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:48.768106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:48.768175 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:48.768274 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:48.769897 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:48.770581 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:48.772509 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:48.773004 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:48.773139 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:48.776851 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:48.776857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:48.776860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:48.776894 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:48.779480 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:48.780244 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:48.780249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:48.780251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:48.780372 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:48.780830 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:48.780840 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:48.780842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:48.781099 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:48.781285 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:48.782007 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 21:41:48.782235 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":314,"thread_start_us":199,"threads_started":1}
I20260926 21:41:48.783565 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:48.765439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39365 (local address 127.0.97.2:52823)
0926 21:41:48.766159 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:48.766202 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:48.767857 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:48.768523 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:48.768532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:48.769077 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:48.769737 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:48.769755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:48.770797 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:48.770801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:48.772354 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:48.772366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:48.773264 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:48.773273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:48.773507 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:48.774239 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:41:48.774262 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:48.776618 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:48.779636 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:48.779646 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:48.779665 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:48.780102 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:48.780507 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:48.780512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:48.780516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:48.780706 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:48.781260 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:48.781269 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:48.781627 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 21:41:48.781905 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 21:41:48.782440 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":302,"threads_started":1}
I20260926 21:41:48.785138 20027 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39365
I20260926 21:41:48.785514 20027 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:48.786245 20027 heartbeater.cc:511] Master 127.0.97.62:39365 requested a full tablet report, sending...
I20260926 21:41:48.787783 19704 ts_manager.cc:194] Registered new tserver with Master: ff748379baee43c2b1348fa20f26d9d7 (127.0.97.2:34257)
I20260926 21:41:48.788628 19704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52823
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 21:41:48.826701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39365
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:49.156463 20033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:49.156751 20033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:49.156845 20033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:49.164395 20033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:49.164512 20033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:49.164685 20033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:41:49.174677 20033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:49.175932 20033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:49.177217 20033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:49.190259 20038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:49.190407 20039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:49.190321 20041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:49.191463 20033 server_base.cc:1034] running on GCE node
I20260926 21:41:49.192059 20033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:49.192965 20033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:49.194155 20033 hybrid_clock.cc:648] HybridClock initialized: now 1790458909194071 us; error 77 us; skew 500 ppm
I20260926 21:41:49.194464 20033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:49 dist-test-slave-lhhq krb5kdc[19659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:49.197577 20033 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 21:41:49.199123 20033 webserver.cc:460] Webserver started at http://127.0.97.3:43171/ using document root <none> and password file <none>
I20260926 21:41:49.199795 20033 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:49.199952 20033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:49.200259 20033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:49.203194 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98"
format_stamp: "Formatted at 2026-09-26 21:41:49 on dist-test-slave-lhhq"
server_key: "8a44a681269c2db1271bca4f9bddfe99"
server_key_iv: "e3e38f7adf1a9ae822b97a6fd3ad1966"
server_key_version: "encryptionkey@0"
I20260926 21:41:49.203913 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98"
format_stamp: "Formatted at 2026-09-26 21:41:49 on dist-test-slave-lhhq"
server_key: "8a44a681269c2db1271bca4f9bddfe99"
server_key_iv: "e3e38f7adf1a9ae822b97a6fd3ad1966"
server_key_version: "encryptionkey@0"
I20260926 21:41:49.208766 20033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 21:41:49.212430 20048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:49.213914 20033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:41:49.214085 20033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98"
format_stamp: "Formatted at 2026-09-26 21:41:49 on dist-test-slave-lhhq"
server_key: "8a44a681269c2db1271bca4f9bddfe99"
server_key_iv: "e3e38f7adf1a9ae822b97a6fd3ad1966"
server_key_version: "encryptionkey@0"
I20260926 21:41:49.214287 20033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:49.246133 20033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:49.250234 20033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:49.250523 20033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:49.251576 20033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:49.253597 20033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:49.253695 20033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:49.253821 20033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:49.253911 20033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:49.258966 19893 heartbeater.cc:503] Master 127.0.97.62:39365 was elected leader, sending a full tablet report...
I20260926 21:41:49.301252 20033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33527
I20260926 21:41:49.301465 20160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33527 every 8 connection(s)
I20260926 21:41:49.302973 20033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:41:49.304126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 20033
I20260926 21:41:49.304325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:41:49.304961 388 external_mini_cluster.cc:1428] Setting key a06e8cab0cb6079b0d31e065b1f7d4b3
Sep 26 21:41:49 dist-test-slave-lhhq krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:41:49.329887 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.309081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58847 (local address 127.0.97.62:39365)
0926 21:41:49.311704 (+ 2623us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.311709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.311726 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:41:49.313295 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.313300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.313408 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.313519 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:49.315858 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.317116 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.319163 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.319669 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.319803 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.324021 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:49.324026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:49.324030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:49.324064 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:49.327046 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:49.327698 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:49.327702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:49.327704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:49.327849 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:49.328220 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:49.328224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:49.328228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:49.328485 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:49.328608 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.329305 (+ 697us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.329546 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2520,"thread_start_us":2414,"threads_started":1}
I20260926 21:41:49.330888 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.308477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39365 (local address 127.0.97.3:58847)
0926 21:41:49.309475 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:49.309538 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:49.312937 (+ 3399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:49.313831 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:49.313845 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:49.314707 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:49.315650 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.315682 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.317287 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.317293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.318978 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.318996 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.319907 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.319916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.320168 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.320955 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:41:49.320989 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:49.323810 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:49.327166 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:49.327172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:49.327185 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:49.327589 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:49.327943 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:41:49.327946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:41:49.327948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:41:49.328126 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:41:49.328586 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:49.328592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:49.328906 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 21:41:49.329132 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 21:41:49.329695 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":1744,"threads_started":1}
I20260926 21:41:49.332262 20161 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39365
I20260926 21:41:49.332583 20161 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:49.333395 20161 heartbeater.cc:511] Master 127.0.97.62:39365 requested a full tablet report, sending...
I20260926 21:41:49.335076 19704 ts_manager.cc:194] Registered new tserver with Master: dd4f5bf1a76f4fa38f5daecbf3ab9e98 (127.0.97.3:33527)
I20260926 21:41:49.335983 19704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58847
I20260926 21:41:49.347877 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:41:49.364895 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.351894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.0.97.62:39365)
0926 21:41:49.352196 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.352205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.352509 (+ 304us) server_negotiation.cc:407] Connection header received
0926 21:41:49.352711 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.352717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.352819 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.352926 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:49.354391 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.355119 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.356401 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.356822 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.356956 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.359468 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:49.359474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:49.359481 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:49.359519 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:49.362209 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:49.362937 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:49.362942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:49.362946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:49.363074 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:49.363475 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:49.363478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:49.363480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:49.363844 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:49.364023 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.364373 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.364550 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 21:41:49.368778 19704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:41:49.375701 19704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:41:49.409678 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.387903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.0.97.3:33527)
0926 21:41:49.399644 (+ 11741us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.399655 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.399702 (+ 47us) server_negotiation.cc:407] Connection header received
0926 21:41:49.399838 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.399850 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.400212 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.401407 (+ 1195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:49.401770 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.403787 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.407031 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.408429 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.408557 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.408782 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.409098 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.409310 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11561,"thread_start_us":11420,"threads_started":1}
I20260926 21:41:49.409708 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.387709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33527 (local address 127.0.0.1:36798)
0926 21:41:49.388505 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:49.388544 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:49.388713 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:49.400541 (+ 11828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:49.400553 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:49.400597 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:49.401141 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.401158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.403959 (+ 2801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.403965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.406819 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.406839 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.408636 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.408642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.408837 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.408889 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 21:41:49.409019 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 21:41:49.409305 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":851,"threads_started":1}
I20260926 21:41:49.412295 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.388891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.0.97.1:35971)
0926 21:41:49.404297 (+ 15406us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.404310 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.404349 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:41:49.404471 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.404483 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.404792 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.406087 (+ 1295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:49.406352 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.407689 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.409849 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.411216 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.411366 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.411595 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.411831 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.411982 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15155,"thread_start_us":16072,"threads_started":1}
I20260926 21:41:49.412333 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.388126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35971 (local address 127.0.0.1:56310)
0926 21:41:49.398918 (+ 10792us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:49.398939 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:49.399066 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:49.405282 (+ 6216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:49.405287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:49.405312 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:49.405820 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.405838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.407814 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.407820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.409628 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.409642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.411434 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.411441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.411629 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.411666 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 21:41:49.411776 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 21:41:49.411912 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10640,"mutex_wait_us":4,"thread_start_us":10358,"threads_started":1}
I20260926 21:41:49.414690 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.389576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34257 (local address 127.0.0.1:39724)
0926 21:41:49.406053 (+ 16477us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:49.406073 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:49.406194 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:49.406786 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:49.406791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:49.406816 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:49.407183 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.407193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.408966 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.408972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.412059 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.412075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.413878 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.413885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.414061 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.414103 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 21:41:49.414206 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 21:41:49.414339 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16344,"mutex_wait_us":3,"thread_start_us":16196,"threads_started":1}
I20260926 21:41:49.414116 20096 tablet_service.cc:1467] Processing CreateTablet for tablet 4a76530250194c20b020adb030dab597 (DEFAULT_TABLE table=test-table [id=34f5edb04fe04d4d922dc6682592d472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:49.415089 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.389628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.0.97.2:34257)
0926 21:41:49.405571 (+ 15943us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.405579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.406183 (+ 604us) server_negotiation.cc:407] Connection header received
0926 21:41:49.406350 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.406357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.406580 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.406770 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:49.407352 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.408819 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.412255 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.413655 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.413788 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.413984 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.414606 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.414783 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15788,"thread_start_us":15560,"threads_started":1}
I20260926 21:41:49.415853 20096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a76530250194c20b020adb030dab597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:49.415899 19828 tablet_service.cc:1467] Processing CreateTablet for tablet 4a76530250194c20b020adb030dab597 (DEFAULT_TABLE table=test-table [id=34f5edb04fe04d4d922dc6682592d472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:49.417498 19828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a76530250194c20b020adb030dab597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:49.418717 19962 tablet_service.cc:1467] Processing CreateTablet for tablet 4a76530250194c20b020adb030dab597 (DEFAULT_TABLE table=test-table [id=34f5edb04fe04d4d922dc6682592d472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:49.420014 19962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a76530250194c20b020adb030dab597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:49.430388 20180 tablet_bootstrap.cc:492] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: Bootstrap starting.
I20260926 21:41:49.432655 20181 tablet_bootstrap.cc:492] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: Bootstrap starting.
I20260926 21:41:49.434347 20182 tablet_bootstrap.cc:492] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: Bootstrap starting.
I20260926 21:41:49.435300 20180 tablet_bootstrap.cc:654] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:49.436573 20181 tablet_bootstrap.cc:654] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:49.436831 20180 log.cc:824] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:49.437373 20182 tablet_bootstrap.cc:654] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:49.438580 20182 log.cc:824] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:49.438632 20181 log.cc:824] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:49.440717 20180 tablet_bootstrap.cc:492] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: No bootstrap required, opened a new log
I20260926 21:41:49.441048 20182 tablet_bootstrap.cc:492] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: No bootstrap required, opened a new log
I20260926 21:41:49.441103 20180 ts_tablet_manager.cc:1397] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 21:41:49.441373 20182 ts_tablet_manager.cc:1397] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260926 21:41:49.447180 20181 tablet_bootstrap.cc:492] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: No bootstrap required, opened a new log
I20260926 21:41:49.447525 20181 ts_tablet_manager.cc:1397] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260926 21:41:49.449112 20182 raft_consensus.cc:348] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } }
I20260926 21:41:49.449563 20182 raft_consensus.cc:374] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:49.449654 20182 raft_consensus.cc:729] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff748379baee43c2b1348fa20f26d9d7, State: Initialized, Role: FOLLOWER
I20260926 21:41:49.450333 20182 consensus_queue.cc:260] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } }
I20260926 21:41:49.451601 20027 heartbeater.cc:503] Master 127.0.97.62:39365 was elected leader, sending a full tablet report...
I20260926 21:41:49.452124 20182 ts_tablet_manager.cc:1428] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 21:41:49.456705 20180 raft_consensus.cc:348] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } }
I20260926 21:41:49.457254 20180 raft_consensus.cc:374] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:49.457391 20180 raft_consensus.cc:729] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4f5bf1a76f4fa38f5daecbf3ab9e98, State: Initialized, Role: FOLLOWER
I20260926 21:41:49.458117 20180 consensus_queue.cc:260] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } }
I20260926 21:41:49.459342 20161 heartbeater.cc:503] Master 127.0.97.62:39365 was elected leader, sending a full tablet report...
I20260926 21:41:49.459575 20181 raft_consensus.cc:348] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } }
I20260926 21:41:49.460098 20181 raft_consensus.cc:374] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:49.460222 20180 ts_tablet_manager.cc:1428] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: Time spent starting tablet: real 0.019s user 0.011s sys 0.004s
I20260926 21:41:49.460232 20181 raft_consensus.cc:729] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da74f1876def430cb2a337a0cda654f3, State: Initialized, Role: FOLLOWER
I20260926 21:41:49.460919 20181 consensus_queue.cc:260] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } }
I20260926 21:41:49.462917 20181 ts_tablet_manager.cc:1428] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260926 21:41:49.477145 19894 tablet.cc:2369] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:41:49.515965 20028 tablet.cc:2369] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:41:49.556402 20162 tablet.cc:2369] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:49.619246 20186 raft_consensus.cc:482] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:41:49.619549 20186 raft_consensus.cc:504] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } }
I20260926 21:41:49.621546 20186 leader_election.cc:288] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd4f5bf1a76f4fa38f5daecbf3ab9e98 (127.0.97.3:33527), da74f1876def430cb2a337a0cda654f3 (127.0.97.1:35971)
I20260926 21:41:49.628858 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.622153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43871 (local address 127.0.97.3:33527)
0926 21:41:49.622368 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.622376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.623128 (+ 752us) server_negotiation.cc:407] Connection header received
0926 21:41:49.623352 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.623358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.623438 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.623505 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:49.624218 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.625236 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.627096 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.627894 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.627989 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.628091 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.628460 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.628596 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:41:49.628880 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.622113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49743 (local address 127.0.97.1:35971)
0926 21:41:49.622333 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.622340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.623175 (+ 835us) server_negotiation.cc:407] Connection header received
0926 21:41:49.623419 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.623423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.623506 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.623599 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:49.624406 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.625338 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.627190 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.627992 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.628128 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.628234 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.628524 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.628651 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:41:49.629034 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.622110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35971 (local address 127.0.97.2:49743)
0926 21:41:49.623034 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:49.623048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:49.623267 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:49.623632 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:49.623638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:49.623676 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:49.624236 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.624248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.625477 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.625482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.627044 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.627059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.628214 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.628218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.628349 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.628370 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 21:41:49.628485 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 21:41:49.628599 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":4,"spinlock_wait_cycles":18176,"thread_start_us":480,"threads_started":1}
I20260926 21:41:49.629443 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.622070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33527 (local address 127.0.97.2:43871)
0926 21:41:49.623000 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:49.623020 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:49.623188 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:49.623567 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:49.623574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:49.623610 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:49.624024 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.624033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.625382 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.625389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.626933 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:49.626944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.628118 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.628124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:49.628304 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.628327 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 21:41:49.628400 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 21:41:49.628532 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"spinlock_wait_cycles":21248,"thread_start_us":628,"threads_started":1}
I20260926 21:41:49.630203 19848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a76530250194c20b020adb030dab597" candidate_uuid: "ff748379baee43c2b1348fa20f26d9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da74f1876def430cb2a337a0cda654f3" is_pre_election: true
I20260926 21:41:49.630412 20116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a76530250194c20b020adb030dab597" candidate_uuid: "ff748379baee43c2b1348fa20f26d9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" is_pre_election: true
I20260926 21:41:49.630842 19848 raft_consensus.cc:2454] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff748379baee43c2b1348fa20f26d9d7 in term 0.
I20260926 21:41:49.630893 20116 raft_consensus.cc:2454] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff748379baee43c2b1348fa20f26d9d7 in term 0.
I20260926 21:41:49.631536 19917 leader_election.cc:302] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: da74f1876def430cb2a337a0cda654f3, dd4f5bf1a76f4fa38f5daecbf3ab9e98, ff748379baee43c2b1348fa20f26d9d7; no voters:
I20260926 21:41:49.631945 20186 raft_consensus.cc:2781] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:41:49.632090 20186 raft_consensus.cc:482] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:41:49.632192 20186 raft_consensus.cc:3037] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:49.634379 20186 raft_consensus.cc:504] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } }
I20260926 21:41:49.634958 20186 leader_election.cc:288] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [CANDIDATE]: Term 1 election: Requested vote from peers dd4f5bf1a76f4fa38f5daecbf3ab9e98 (127.0.97.3:33527), da74f1876def430cb2a337a0cda654f3 (127.0.97.1:35971)
I20260926 21:41:49.635385 20116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a76530250194c20b020adb030dab597" candidate_uuid: "ff748379baee43c2b1348fa20f26d9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98"
I20260926 21:41:49.635442 19848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a76530250194c20b020adb030dab597" candidate_uuid: "ff748379baee43c2b1348fa20f26d9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da74f1876def430cb2a337a0cda654f3"
I20260926 21:41:49.635617 20116 raft_consensus.cc:3037] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:49.635669 19848 raft_consensus.cc:3037] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:49.637903 19848 raft_consensus.cc:2454] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff748379baee43c2b1348fa20f26d9d7 in term 1.
I20260926 21:41:49.637904 20116 raft_consensus.cc:2454] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff748379baee43c2b1348fa20f26d9d7 in term 1.
I20260926 21:41:49.638278 19917 leader_election.cc:302] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da74f1876def430cb2a337a0cda654f3, ff748379baee43c2b1348fa20f26d9d7; no voters:
I20260926 21:41:49.638559 20186 raft_consensus.cc:2781] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:49.638983 20186 raft_consensus.cc:686] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 1 LEADER]: Becoming Leader. State: Replica: ff748379baee43c2b1348fa20f26d9d7, State: Running, Role: LEADER
I20260926 21:41:49.639545 20186 consensus_queue.cc:237] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } }
I20260926 21:41:49.643970 19704 catalog_manager.cc:5495] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff748379baee43c2b1348fa20f26d9d7 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ff748379baee43c2b1348fa20f26d9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff748379baee43c2b1348fa20f26d9d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 } health_report { overall_health: UNKNOWN } } }
I20260926 21:41:49.694171 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.688652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.0.97.2:34257)
0926 21:41:49.688883 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.688888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.688907 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:41:49.689006 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.689009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.689089 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.689302 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:41:49.689604 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.690346 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.691844 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.692276 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.692382 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.692852 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:41:49.693513 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:41:49.693673 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.693796 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.693862 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 21:41:49.702440 20116 raft_consensus.cc:1261] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [term 1 FOLLOWER]: Refusing update from remote peer ff748379baee43c2b1348fa20f26d9d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:49.702450 19848 raft_consensus.cc:1261] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [term 1 FOLLOWER]: Refusing update from remote peer ff748379baee43c2b1348fa20f26d9d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:49.703382 20191 consensus_queue.cc:1035] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4f5bf1a76f4fa38f5daecbf3ab9e98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:49.703758 20186 consensus_queue.cc:1035] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da74f1876def430cb2a337a0cda654f3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:49.714949 20194 mvcc.cc:205] Tried to move back new op lower bound from 7333719694131621888 to 7333719693887401984. Current Snapshot: MvccSnapshot[applied={T|T < 7333719694131621888}]
I20260926 21:41:49.730105 19703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:49.730481 19703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:49.734512 19703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a628efc2b2d4c40aba7296b5467b17b: Sending DeleteTablet for 3 replicas of tablet 4a76530250194c20b020adb030dab597
I20260926 21:41:49.735854 20096 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a76530250194c20b020adb030dab597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36798
I20260926 21:41:49.735875 19962 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a76530250194c20b020adb030dab597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39724
I20260926 21:41:49.735878 19828 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a76530250194c20b020adb030dab597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56310
I20260926 21:41:49.736917 20204 tablet_replica.cc:330] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: stopping tablet replica
I20260926 21:41:49.737609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 19765
I20260926 21:41:49.737609 20204 raft_consensus.cc:2229] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:49.738400 20204 raft_consensus.cc:2258] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:49.739310 20205 tablet_replica.cc:330] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: stopping tablet replica
I20260926 21:41:49.739773 20205 raft_consensus.cc:2229] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:49.740294 20205 raft_consensus.cc:2258] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:49.742616 20205 ts_tablet_manager.cc:1905] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:49.748010 20204 ts_tablet_manager.cc:1905] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:49.750065 20205 ts_tablet_manager.cc:1918] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:49.750334 20205 log.cc:1196] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/4a76530250194c20b020adb030dab597
I20260926 21:41:49.750934 20205 ts_tablet_manager.cc:1939] T 4a76530250194c20b020adb030dab597 P dd4f5bf1a76f4fa38f5daecbf3ab9e98: Deleting consensus metadata
I20260926 21:41:49.752949 19691 catalog_manager.cc:4841] TS dd4f5bf1a76f4fa38f5daecbf3ab9e98 (127.0.97.3:33527): tablet 4a76530250194c20b020adb030dab597 (table test-table [id=34f5edb04fe04d4d922dc6682592d472]) successfully deleted
I20260926 21:41:49.754503 20206 tablet_replica.cc:330] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: stopping tablet replica
I20260926 21:41:49.755029 20206 raft_consensus.cc:2229] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:41:49.755779 20206 raft_consensus.cc:2258] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:49.761366 20204 ts_tablet_manager.cc:1918] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:49.761559 20204 log.cc:1196] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/4a76530250194c20b020adb030dab597
Sep 26 21:41:49 dist-test-slave-lhhq krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 21:41:49.762120 20204 ts_tablet_manager.cc:1939] T 4a76530250194c20b020adb030dab597 P da74f1876def430cb2a337a0cda654f3: Deleting consensus metadata
I20260926 21:41:49.764358 19690 catalog_manager.cc:4841] TS da74f1876def430cb2a337a0cda654f3 (127.0.97.1:35971): tablet 4a76530250194c20b020adb030dab597 (table test-table [id=34f5edb04fe04d4d922dc6682592d472]) successfully deleted
I20260926 21:41:49.766688 20206 ts_tablet_manager.cc:1905] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:49.773015 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.739228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.0.97.1:35971)
0926 21:41:49.739455 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.739464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.749472 (+ 10008us) server_negotiation.cc:407] Connection header received
0926 21:41:49.750139 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.750150 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.750274 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.750379 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:49.752812 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.754213 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.756245 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.756956 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.757076 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.763102 (+ 6026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:49.763120 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:49.763137 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:49.763207 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:49.768682 (+ 5475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:49.770181 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:49.770215 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:49.770220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:49.770407 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:49.770906 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:49.770912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:49.770916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:49.771387 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:49.771679 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.772325 (+ 646us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.772581 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:41:49.774722 19888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:49.778055 20206 ts_tablet_manager.cc:1918] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:49.778285 20206 log.cc:1196] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/4a76530250194c20b020adb030dab597
I20260926 21:41:49.778992 20206 ts_tablet_manager.cc:1939] T 4a76530250194c20b020adb030dab597 P ff748379baee43c2b1348fa20f26d9d7: Deleting consensus metadata
I20260926 21:41:49.781343 19691 catalog_manager.cc:4841] TS ff748379baee43c2b1348fa20f26d9d7 (127.0.97.2:34257): tablet 4a76530250194c20b020adb030dab597 (table test-table [id=34f5edb04fe04d4d922dc6682592d472]) successfully deleted
I20260926 21:41:49.873907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 19765
I20260926 21:41:49.892073 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 19899
Sep 26 21:41:49 dist-test-slave-lhhq krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 21:41:49.910676 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:49.892974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.0.97.2:34257)
0926 21:41:49.893166 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:49.893171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:49.893247 (+ 76us) server_negotiation.cc:407] Connection header received
0926 21:41:49.893413 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:49.893418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:49.893493 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:49.893604 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:49.895279 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.896252 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.898355 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:49.898946 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:49.899068 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:49.904248 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:49.904264 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:49.904281 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:49.904324 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:49.907553 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:49.908279 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:49.908288 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:49.908290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:49.908440 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:49.908893 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:49.908902 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:49.908904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:49.909358 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:49.909725 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:49.910057 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 21:41:49.910297 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:41:49.911409 20022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:50.027051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 19899
I20260926 21:41:50.046033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20033
Sep 26 21:41:50 dist-test-slave-lhhq krb5kdc[19659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 21:41:50.061414 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:50.046830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.0.97.3:33527)
0926 21:41:50.047022 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:50.047027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:50.047047 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:41:50.047268 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:50.047273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:50.047363 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:50.047463 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:50.048803 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:50.049637 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:50.051054 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:50.051466 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:50.051573 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:50.056046 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:50.056060 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:50.056077 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:50.056112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:50.058851 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:50.059525 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:50.059532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:50.059534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:50.059661 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:50.060044 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:50.060047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:50.060049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:50.060387 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:50.060592 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:50.060922 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 21:41:50.061076 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:41:50.062072 20156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:50.151844 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 20033
I20260926 21:41:50.169099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 19672
I20260926 21:41:50.170286 19734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:50.282760 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 19672
2026-09-26T21:41:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4499 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T21:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:41:50Z Disabled control of system clock
I20260926 21:41:50.359064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37785
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:41:50.692150 20215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:50.692451 20215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:50.692533 20215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:50.699846 20215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:50.699975 20215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:50.700013 20215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:50.700062 20215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:50.700106 20215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:50.709568 20215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:50.710830 20215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:50.712688 20215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:50.724792 20220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:50.724750 20223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:50.724722 20221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:50.725641 20215 server_base.cc:1034] running on GCE node
I20260926 21:41:50.726523 20215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:50.728030 20215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:50.729318 20215 hybrid_clock.cc:648] HybridClock initialized: now 1790458910729236 us; error 54 us; skew 500 ppm
I20260926 21:41:50.729727 20215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:50.740738 20215 webserver.cc:460] Webserver started at http://127.0.97.62:41753/ using document root <none> and password file <none>
I20260926 21:41:50.741670 20215 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:50.741819 20215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:50.742170 20215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:50.746928 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "cc5a2de60c77444584efd5f452c23738"
format_stamp: "Formatted at 2026-09-26 21:41:50 on dist-test-slave-lhhq"
server_key: "111d0dc8af3b8dd86125efece815cde7"
server_key_iv: "57bf2b7a380afe264e022da0297367b4"
server_key_version: "encryptionkey@0"
I20260926 21:41:50.748106 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "cc5a2de60c77444584efd5f452c23738"
format_stamp: "Formatted at 2026-09-26 21:41:50 on dist-test-slave-lhhq"
server_key: "111d0dc8af3b8dd86125efece815cde7"
server_key_iv: "57bf2b7a380afe264e022da0297367b4"
server_key_version: "encryptionkey@0"
I20260926 21:41:50.754540 20215 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 21:41:50.758052 20229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:50.759457 20215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:41:50.759648 20215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "cc5a2de60c77444584efd5f452c23738"
format_stamp: "Formatted at 2026-09-26 21:41:50 on dist-test-slave-lhhq"
server_key: "111d0dc8af3b8dd86125efece815cde7"
server_key_iv: "57bf2b7a380afe264e022da0297367b4"
server_key_version: "encryptionkey@0"
I20260926 21:41:50.759883 20215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:50.771605 20215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:50.772495 20215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:50.772789 20215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:50.792299 20215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37785
I20260926 21:41:50.792306 20280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37785 every 8 connection(s)
I20260926 21:41:50.793951 20215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:50.798233 20281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:50.804133 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 20215
I20260926 21:41:50.804335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:41:50.804798 388 external_mini_cluster.cc:1428] Setting key 3b3727e28511a7f24b0fc5c6c23fe7cd
I20260926 21:41:50.810494 20281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738: Bootstrap starting.
I20260926 21:41:50.814476 20281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:50.815848 20281 log.cc:824] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:50.819650 20281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738: No bootstrap required, opened a new log
I20260926 21:41:50.820787 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:50.807221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.0.97.62:37785)
0926 21:41:50.808487 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:50.808499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:50.808546 (+ 47us) server_negotiation.cc:407] Connection header received
0926 21:41:50.810467 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:50.810497 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:50.811104 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:50.811473 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:50.812230 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:50.813653 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:50.815687 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:50.816696 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:50.816951 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:50.818198 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:50.818206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:50.818223 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:50.818823 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:50.818932 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:50.818961 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:50.819057 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:50.819320 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 21:41:50.819693 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":518,"threads_started":1}
I20260926 21:41:50.828011 20281 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5a2de60c77444584efd5f452c23738" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } }
I20260926 21:41:50.828461 20281 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:50.828579 20281 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5a2de60c77444584efd5f452c23738, State: Initialized, Role: FOLLOWER
I20260926 21:41:50.829155 20281 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5a2de60c77444584efd5f452c23738" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } }
I20260926 21:41:50.829429 20281 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:50.829535 20281 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:50.829664 20281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:50.832104 20281 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5a2de60c77444584efd5f452c23738" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } }
I20260926 21:41:50.832777 20281 leader_election.cc:302] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc5a2de60c77444584efd5f452c23738; no voters:
I20260926 21:41:50.833515 20281 leader_election.cc:288] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:41:50.833721 20286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:50.834542 20286 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [term 1 LEADER]: Becoming Leader. State: Replica: cc5a2de60c77444584efd5f452c23738, State: Running, Role: LEADER
I20260926 21:41:50.835104 20286 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5a2de60c77444584efd5f452c23738" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } }
I20260926 21:41:50.835796 20281 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:50.838207 20287 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc5a2de60c77444584efd5f452c23738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5a2de60c77444584efd5f452c23738" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } } }
I20260926 21:41:50.838207 20288 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc5a2de60c77444584efd5f452c23738. Latest consensus state: current_term: 1 leader_uuid: "cc5a2de60c77444584efd5f452c23738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5a2de60c77444584efd5f452c23738" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } } }
I20260926 21:41:50.838856 20287 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:50.840695 20288 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:50.842113 20294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:50.847510 20294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:50.857127 20294 catalog_manager.cc:1347] Generated new cluster ID: 082b914bf4a74358b5856bb37da5e6ff
I20260926 21:41:50.857295 20294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:50.875943 20294 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:41:50.877357 20294 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:50.885612 20294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738: Generated new TSK 0
I20260926 21:41:50.886528 20294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:41:50.895468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--builtin_ntp_servers=127.0.97.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 21:41:51.229370 20305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:51.229652 20305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:51.229823 20305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:51.237108 20305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:51.237272 20305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:51.237462 20305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 21:41:51.246981 20305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:51.248381 20305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:51.250231 20305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:51.264472 20313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:51.264504 20311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:51.264546 20310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:51.265779 20305 server_base.cc:1034] running on GCE node
I20260926 21:41:51.266494 20305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:51.267493 20305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:51.268726 20305 hybrid_clock.cc:648] HybridClock initialized: now 1790458911268669 us; error 51 us; skew 500 ppm
I20260926 21:41:51.269066 20305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:51.271492 20305 webserver.cc:460] Webserver started at http://127.0.97.1:40235/ using document root <none> and password file <none>
I20260926 21:41:51.272205 20305 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:51.272338 20305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:51.272662 20305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:51.275610 20305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/data/instance:
uuid: "8dda73b9144e4033bde864d0c4d1c54d"
format_stamp: "Formatted at 2026-09-26 21:41:51 on dist-test-slave-lhhq"
server_key: "006064e93f91cdaa238756f490d34052"
server_key_iv: "420af337409d36fb9157ff8d98e0450c"
server_key_version: "encryptionkey@0"
I20260926 21:41:51.276425 20305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8dda73b9144e4033bde864d0c4d1c54d"
format_stamp: "Formatted at 2026-09-26 21:41:51 on dist-test-slave-lhhq"
server_key: "006064e93f91cdaa238756f490d34052"
server_key_iv: "420af337409d36fb9157ff8d98e0450c"
server_key_version: "encryptionkey@0"
I20260926 21:41:51.281653 20305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 21:41:51.285281 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:51.286662 20305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:41:51.286850 20305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/wal
uuid: "8dda73b9144e4033bde864d0c4d1c54d"
format_stamp: "Formatted at 2026-09-26 21:41:51 on dist-test-slave-lhhq"
server_key: "006064e93f91cdaa238756f490d34052"
server_key_iv: "420af337409d36fb9157ff8d98e0450c"
server_key_version: "encryptionkey@0"
I20260926 21:41:51.287075 20305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:51.321851 20305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:51.322749 20305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:51.323047 20305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:51.324098 20305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:51.325882 20305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:51.326004 20305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:51.326131 20305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:51.326217 20305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:51.360766 20305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37599
I20260926 21:41:51.360836 20431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37599 every 8 connection(s)
I20260926 21:41:51.362237 20305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/data/info.pb
I20260926 21:41:51.370741 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 20305
I20260926 21:41:51.370905 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/wal/instance
I20260926 21:41:51.371352 388 external_mini_cluster.cc:1428] Setting key 2a4a4ec315bbe78009ad7cdebaf96a78
I20260926 21:41:51.374495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--builtin_ntp_servers=127.0.97.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:41:51.376394 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:51.364976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41401 (local address 127.0.97.62:37785)
0926 21:41:51.365494 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:51.365498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:51.366734 (+ 1236us) server_negotiation.cc:407] Connection header received
0926 21:41:51.368654 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:51.368660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:51.368777 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:51.368911 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:51.370123 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:51.370974 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:51.372992 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:51.373593 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:51.373724 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:51.375317 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:51.375327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:51.375336 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:51.375363 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:51.375423 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:51.375444 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:51.375546 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:51.375816 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 21:41:51.375949 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":303,"threads_started":1}
I20260926 21:41:51.377441 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:51.365642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37785 (local address 127.0.97.1:41401)
0926 21:41:51.366557 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:51.366594 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:51.368362 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:51.369174 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:51.369228 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:51.369344 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:51.369946 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:51.369965 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:51.371147 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:51.371151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:51.372833 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:51.372859 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:51.373845 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:51.373850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:51.374061 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:51.374955 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:41:51.374958 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:51.375003 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:41:51.375094 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:51.375576 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:51.375587 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:51.375594 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 21:41:51.375770 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 21:41:51.376118 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":277,"threads_started":1}
I20260926 21:41:51.379231 20432 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37785
I20260926 21:41:51.379599 20432 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:51.380609 20432 heartbeater.cc:511] Master 127.0.97.62:37785 requested a full tablet report, sending...
I20260926 21:41:51.383071 20246 ts_manager.cc:194] Registered new tserver with Master: 8dda73b9144e4033bde864d0c4d1c54d (127.0.97.1:37599)
I20260926 21:41:51.385504 20246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:41401
W20260926 21:41:51.714653 20436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:51.714972 20436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:51.715066 20436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:51.722913 20436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:51.723047 20436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:51.723253 20436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 21:41:51.734753 20436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:51.736121 20436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:51.738027 20436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:51.751396 20441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:51.751425 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:51.751585 20442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:51.752044 20436 server_base.cc:1034] running on GCE node
I20260926 21:41:51.752682 20436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:51.753760 20436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:51.754992 20436 hybrid_clock.cc:648] HybridClock initialized: now 1790458911754940 us; error 46 us; skew 500 ppm
I20260926 21:41:51.755316 20436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:51.757720 20436 webserver.cc:460] Webserver started at http://127.0.97.2:45779/ using document root <none> and password file <none>
I20260926 21:41:51.758486 20436 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:51.758625 20436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:51.758965 20436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:51.761983 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/data/instance:
uuid: "0616810c8fab4690a8e47cc5d6b60755"
format_stamp: "Formatted at 2026-09-26 21:41:51 on dist-test-slave-lhhq"
server_key: "a6bab807eed72321de9fa6b42cbc25e9"
server_key_iv: "a225fea0620089ff91496b20eb610e9d"
server_key_version: "encryptionkey@0"
I20260926 21:41:51.762763 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0616810c8fab4690a8e47cc5d6b60755"
format_stamp: "Formatted at 2026-09-26 21:41:51 on dist-test-slave-lhhq"
server_key: "a6bab807eed72321de9fa6b42cbc25e9"
server_key_iv: "a225fea0620089ff91496b20eb610e9d"
server_key_version: "encryptionkey@0"
I20260926 21:41:51.768327 20436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:41:51.772358 20450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:51.774062 20436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:41:51.774251 20436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/wal
uuid: "0616810c8fab4690a8e47cc5d6b60755"
format_stamp: "Formatted at 2026-09-26 21:41:51 on dist-test-slave-lhhq"
server_key: "a6bab807eed72321de9fa6b42cbc25e9"
server_key_iv: "a225fea0620089ff91496b20eb610e9d"
server_key_version: "encryptionkey@0"
I20260926 21:41:51.774509 20436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:51.789713 20436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:51.790701 20436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:51.790994 20436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:51.792268 20436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:51.793939 20436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:51.794030 20436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:51.794186 20436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:51.794255 20436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:51.830161 20436 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33563
I20260926 21:41:51.830214 20562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33563 every 8 connection(s)
I20260926 21:41:51.831496 20436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/data/info.pb
I20260926 21:41:51.833146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 20436
I20260926 21:41:51.833361 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/wal/instance
I20260926 21:41:51.833829 388 external_mini_cluster.cc:1428] Setting key 8c90922dc4fd090bf4b58c9e06960fc3
I20260926 21:41:51.836642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--builtin_ntp_servers=127.0.97.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 21:41:51.844671 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:51.834326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52301 (local address 127.0.97.62:37785)
0926 21:41:51.835329 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:51.835336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:51.836074 (+ 738us) server_negotiation.cc:407] Connection header received
0926 21:41:51.838157 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:51.838162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:51.838256 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:51.838366 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:51.839402 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:51.840138 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:51.841840 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:51.842297 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:51.842429 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:51.843851 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:51.843862 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:51.843865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:51.843890 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:51.843920 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:51.843932 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:51.844027 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:51.844253 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 21:41:51.844351 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838}
I20260926 21:41:51.845824 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:51.834894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37785 (local address 127.0.97.2:52301)
0926 21:41:51.835915 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:51.835962 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:51.837897 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:51.838607 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:51.838615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:51.838702 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:51.839238 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:51.839254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:51.840295 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:51.840300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:51.841694 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:51.841706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:51.842519 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:51.842526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:51.842808 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:51.843543 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:41:51.843546 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:51.843586 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:41:51.843660 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:51.844027 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:51.844039 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:51.844049 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 21:41:51.844229 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 21:41:51.844634 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":382,"threads_started":1}
I20260926 21:41:51.847332 20563 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37785
I20260926 21:41:51.847643 20563 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:51.848440 20563 heartbeater.cc:511] Master 127.0.97.62:37785 requested a full tablet report, sending...
I20260926 21:41:51.849977 20246 ts_manager.cc:194] Registered new tserver with Master: 0616810c8fab4690a8e47cc5d6b60755 (127.0.97.2:33563)
I20260926 21:41:51.851212 20246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:52301
W20260926 21:41:52.172323 20566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:52.172621 20566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:52.172709 20566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:52.180147 20566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:52.180260 20566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:52.180431 20566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 21:41:52.189705 20566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:52.191134 20566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:52.193009 20566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:52.207214 20571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:52.207191 20572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:52.207366 20574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:52.208103 20566 server_base.cc:1034] running on GCE node
I20260926 21:41:52.208734 20566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:52.209797 20566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:52.211025 20566 hybrid_clock.cc:648] HybridClock initialized: now 1790458912210987 us; error 51 us; skew 500 ppm
I20260926 21:41:52.211345 20566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 21:41:52.213702 20566 webserver.cc:460] Webserver started at http://127.0.97.3:35437/ using document root <none> and password file <none>
I20260926 21:41:52.214413 20566 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:52.214547 20566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:52.214870 20566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:52.217805 20566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/data/instance:
uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8"
format_stamp: "Formatted at 2026-09-26 21:41:52 on dist-test-slave-lhhq"
server_key: "9f29dff1cb21d29f4905382c12bd5f78"
server_key_iv: "2d67161c06d6abebd733a2e158b80390"
server_key_version: "encryptionkey@0"
I20260926 21:41:52.218601 20566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8"
format_stamp: "Formatted at 2026-09-26 21:41:52 on dist-test-slave-lhhq"
server_key: "9f29dff1cb21d29f4905382c12bd5f78"
server_key_iv: "2d67161c06d6abebd733a2e158b80390"
server_key_version: "encryptionkey@0"
I20260926 21:41:52.223941 20566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:41:52.228039 20580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:52.229465 20566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:41:52.229640 20566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/wal
uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8"
format_stamp: "Formatted at 2026-09-26 21:41:52 on dist-test-slave-lhhq"
server_key: "9f29dff1cb21d29f4905382c12bd5f78"
server_key_iv: "2d67161c06d6abebd733a2e158b80390"
server_key_version: "encryptionkey@0"
I20260926 21:41:52.229862 20566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:52.244051 20566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:52.244915 20566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:52.245265 20566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:52.246270 20566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 21:41:52.247838 20566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 21:41:52.247964 20566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:52.248111 20566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 21:41:52.248201 20566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:52.296442 20566 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44065
I20260926 21:41:52.296501 20692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44065 every 8 connection(s)
I20260926 21:41:52.297863 20566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/data/info.pb
I20260926 21:41:52.306295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 20566
I20260926 21:41:52.306463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/wal/instance
I20260926 21:41:52.306948 388 external_mini_cluster.cc:1428] Setting key b503f5dbe10bf8b5632f120638977552
I20260926 21:41:52.311097 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.300512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49605 (local address 127.0.97.62:37785)
0926 21:41:52.300732 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.300737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.302135 (+ 1398us) server_negotiation.cc:407] Connection header received
0926 21:41:52.303942 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.303947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.304043 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.304156 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:52.305381 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.306162 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.308134 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.308785 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.308929 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.310450 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:52.310455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:52.310458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:52.310475 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:52.310505 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:52.310516 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:52.310587 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.310777 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.310851 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 21:41:52.312059 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.301013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37785 (local address 127.0.97.3:49605)
0926 21:41:52.301969 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:52.302007 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:52.303667 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:52.304407 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:52.304416 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:52.304507 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:41:52.305160 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.305187 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.306323 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.306328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.307987 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.308002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.309025 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.309033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.309347 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.310151 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 21:41:52.310154 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:41:52.310192 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 21:41:52.310282 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:41:52.310601 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:41:52.310608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:41:52.310614 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 21:41:52.310741 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 21:41:52.311024 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":338,"threads_started":1}
I20260926 21:41:52.313449 20693 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37785
I20260926 21:41:52.313764 20693 heartbeater.cc:464] Registering TS with master...
I20260926 21:41:52.314512 20693 heartbeater.cc:511] Master 127.0.97.62:37785 requested a full tablet report, sending...
I20260926 21:41:52.315932 20246 ts_manager.cc:194] Registered new tserver with Master: ed4a1846e8af4c5ca9d6d07fb51472f8 (127.0.97.3:44065)
I20260926 21:41:52.316794 20246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49605
I20260926 21:41:52.322201 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 21:41:52.332219 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.326276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.0.97.62:37785)
0926 21:41:52.326477 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.326486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.326847 (+ 361us) server_negotiation.cc:407] Connection header received
0926 21:41:52.327039 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.327044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.327134 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.327238 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:52.327713 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.328412 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.329823 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.330256 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.330398 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.331491 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:52.331497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:52.331501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:52.331528 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:52.331569 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:52.331581 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:52.331676 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.331817 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.331919 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:41:52.336256 20246 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 21:41:52.343487 20246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 21:41:52.368387 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.356644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.97.1:37599)
0926 21:41:52.357576 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.357587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.358232 (+ 645us) server_negotiation.cc:407] Connection header received
0926 21:41:52.358349 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.358356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.358620 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.358878 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:52.360080 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.361808 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.364638 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.366199 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.366316 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.366581 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.367722 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.367911 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":652,"threads_started":1}
I20260926 21:41:52.369863 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.355973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37599 (local address 127.0.0.1:33956)
0926 21:41:52.357941 (+ 1968us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:52.357955 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:52.358098 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:52.359184 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:52.359192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:52.359227 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:52.359797 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.359812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.361989 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.361994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.364359 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.364374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.366722 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.366728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.366975 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.367027 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 21:41:52.367158 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 21:41:52.367351 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1861,"thread_start_us":1723,"threads_started":1}
I20260926 21:41:52.372130 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.355940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44065 (local address 127.0.0.1:43868)
0926 21:41:52.356749 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:52.356802 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:52.356972 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:52.358205 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:52.358216 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:52.358261 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:52.358750 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.358768 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.364430 (+ 5662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.364439 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.368924 (+ 4485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.368947 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.371517 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.371523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.371675 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.371701 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 21:41:52.371763 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 21:41:52.371860 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":827,"threads_started":1}
I20260926 21:41:52.373332 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.356463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.0.97.3:44065)
0926 21:41:52.357466 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.357482 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.357520 (+ 38us) server_negotiation.cc:407] Connection header received
0926 21:41:52.357652 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.357663 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.357952 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.360351 (+ 2399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:52.360715 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.364236 (+ 3521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.369279 (+ 5043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.371280 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.372175 (+ 895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.372515 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.372594 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.372771 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":920,"threads_started":1}
I20260926 21:41:52.375268 20367 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7 (DEFAULT_TABLE table=test-table [id=74030a5ccbb54fa4bc98b1f677c03c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:52.376782 20367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:52.376576 20628 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7 (DEFAULT_TABLE table=test-table [id=74030a5ccbb54fa4bc98b1f677c03c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:52.378082 20628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:52.378922 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.358087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33563 (local address 127.0.0.1:37036)
0926 21:41:52.370449 (+ 12362us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:52.370475 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:52.370610 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:52.372211 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:52.372215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:52.372246 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:52.372428 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.372439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.374548 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.374553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.376262 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.376276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.378110 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.378116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.378317 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.378350 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 21:41:52.378449 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 21:41:52.378614 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12195,"thread_start_us":12758,"threads_started":1}
I20260926 21:41:52.379072 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.358360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.0.97.2:33563)
0926 21:41:52.371582 (+ 13222us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.371591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.371619 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:41:52.371722 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.371729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.371988 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.372617 (+ 629us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:52.372839 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.374384 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.376428 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.377933 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.378079 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.378320 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.378514 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.378715 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13066,"thread_start_us":12936,"threads_started":1}
I20260926 21:41:52.383179 20498 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7 (DEFAULT_TABLE table=test-table [id=74030a5ccbb54fa4bc98b1f677c03c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 21:41:52.384521 20498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:52.388856 20432 heartbeater.cc:503] Master 127.0.97.62:37785 was elected leader, sending a full tablet report...
I20260926 21:41:52.389928 20711 tablet_bootstrap.cc:492] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: Bootstrap starting.
I20260926 21:41:52.393178 20712 tablet_bootstrap.cc:492] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: Bootstrap starting.
I20260926 21:41:52.393836 20711 tablet_bootstrap.cc:654] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:52.395171 20711 log.cc:824] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:52.397270 20712 tablet_bootstrap.cc:654] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:52.398478 20712 log.cc:824] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:52.398936 20714 tablet_bootstrap.cc:492] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: Bootstrap starting.
I20260926 21:41:52.399685 20711 tablet_bootstrap.cc:492] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: No bootstrap required, opened a new log
I20260926 21:41:52.400106 20711 ts_tablet_manager.cc:1397] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 21:41:52.402287 20714 tablet_bootstrap.cc:654] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:52.403270 20712 tablet_bootstrap.cc:492] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: No bootstrap required, opened a new log
I20260926 21:41:52.403256 20714 log.cc:824] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:52.403532 20712 ts_tablet_manager.cc:1397] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 21:41:52.405807 20714 tablet_bootstrap.cc:492] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: No bootstrap required, opened a new log
I20260926 21:41:52.406066 20714 ts_tablet_manager.cc:1397] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 21:41:52.410910 20711 raft_consensus.cc:348] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } }
I20260926 21:41:52.411414 20711 raft_consensus.cc:374] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:52.411547 20711 raft_consensus.cc:729] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4a1846e8af4c5ca9d6d07fb51472f8, State: Initialized, Role: FOLLOWER
I20260926 21:41:52.412253 20711 consensus_queue.cc:260] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } }
I20260926 21:41:52.413515 20693 heartbeater.cc:503] Master 127.0.97.62:37785 was elected leader, sending a full tablet report...
I20260926 21:41:52.414206 20711 ts_tablet_manager.cc:1428] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260926 21:41:52.414440 20714 raft_consensus.cc:348] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } }
I20260926 21:41:52.414527 20712 raft_consensus.cc:348] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } }
I20260926 21:41:52.414911 20712 raft_consensus.cc:374] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:52.414899 20714 raft_consensus.cc:374] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:52.415004 20714 raft_consensus.cc:729] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0616810c8fab4690a8e47cc5d6b60755, State: Initialized, Role: FOLLOWER
I20260926 21:41:52.415048 20712 raft_consensus.cc:729] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dda73b9144e4033bde864d0c4d1c54d, State: Initialized, Role: FOLLOWER
I20260926 21:41:52.415625 20712 consensus_queue.cc:260] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } }
I20260926 21:41:52.415820 20714 consensus_queue.cc:260] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } }
I20260926 21:41:52.417754 20712 ts_tablet_manager.cc:1428] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 21:41:52.419055 20563 heartbeater.cc:503] Master 127.0.97.62:37785 was elected leader, sending a full tablet report...
I20260926 21:41:52.419665 20714 ts_tablet_manager.cc:1428] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260926 21:41:52.529958 20717 raft_consensus.cc:482] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 21:41:52.530347 20717 raft_consensus.cc:504] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } }
I20260926 21:41:52.532274 20717 leader_election.cc:288] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dda73b9144e4033bde864d0c4d1c54d (127.0.97.1:37599), 0616810c8fab4690a8e47cc5d6b60755 (127.0.97.2:33563)
I20260926 21:41:52.539244 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.532997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48485 (local address 127.0.97.1:37599)
0926 21:41:52.533269 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.533276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.533302 (+ 26us) server_negotiation.cc:407] Connection header received
0926 21:41:52.533390 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.533398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.533505 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.533624 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:52.534326 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.535386 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.537421 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.538228 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.538386 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.538498 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.538825 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.538962 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 21:41:52.539283 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.532734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37599 (local address 127.0.97.3:48485)
0926 21:41:52.532952 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:52.532977 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:52.533155 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:52.533828 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:52.533835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:52.533869 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:52.534117 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.534132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.535518 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.535527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.537238 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.537269 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.538395 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.538401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.538598 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.538623 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 21:41:52.538797 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 21:41:52.538943 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":29}
I20260926 21:41:52.539307 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.533424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44975 (local address 127.0.97.2:33563)
0926 21:41:52.533676 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.533696 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.534089 (+ 393us) server_negotiation.cc:407] Connection header received
0926 21:41:52.534251 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.534256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.534333 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.534411 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 21:41:52.535076 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.535733 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.537682 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.538416 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.538556 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.538644 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.538926 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.539052 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 21:41:52.539404 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.533158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33563 (local address 127.0.97.3:44975)
0926 21:41:52.533940 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 21:41:52.533959 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:41:52.534092 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:41:52.534458 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:41:52.534464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:41:52.534493 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 21:41:52.534903 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.534916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.535941 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.535947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.537505 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:41:52.537517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.538587 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.538593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:41:52.538761 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.538783 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 21:41:52.538879 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 21:41:52.539000 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":42,"thread_start_us":419,"threads_started":1}
I20260926 21:41:52.540441 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9b5a5223ac4b7da2c40ab386eeb9b7" candidate_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0616810c8fab4690a8e47cc5d6b60755" is_pre_election: true
I20260926 21:41:52.540705 20387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9b5a5223ac4b7da2c40ab386eeb9b7" candidate_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dda73b9144e4033bde864d0c4d1c54d" is_pre_election: true
I20260926 21:41:52.541028 20518 raft_consensus.cc:2454] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4a1846e8af4c5ca9d6d07fb51472f8 in term 0.
I20260926 21:41:52.541324 20387 raft_consensus.cc:2454] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4a1846e8af4c5ca9d6d07fb51472f8 in term 0.
I20260926 21:41:52.541808 20582 leader_election.cc:302] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0616810c8fab4690a8e47cc5d6b60755, ed4a1846e8af4c5ca9d6d07fb51472f8; no voters:
I20260926 21:41:52.542219 20717 raft_consensus.cc:2781] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 21:41:52.542407 20717 raft_consensus.cc:482] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 21:41:52.542532 20717 raft_consensus.cc:3037] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:52.544994 20717 raft_consensus.cc:504] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } }
I20260926 21:41:52.545744 20717 leader_election.cc:288] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8dda73b9144e4033bde864d0c4d1c54d (127.0.97.1:37599), 0616810c8fab4690a8e47cc5d6b60755 (127.0.97.2:33563)
I20260926 21:41:52.546099 20387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9b5a5223ac4b7da2c40ab386eeb9b7" candidate_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dda73b9144e4033bde864d0c4d1c54d"
I20260926 21:41:52.546286 20387 raft_consensus.cc:3037] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:52.546275 20518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9b5a5223ac4b7da2c40ab386eeb9b7" candidate_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0616810c8fab4690a8e47cc5d6b60755"
I20260926 21:41:52.546489 20518 raft_consensus.cc:3037] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:52.548811 20387 raft_consensus.cc:2454] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4a1846e8af4c5ca9d6d07fb51472f8 in term 1.
I20260926 21:41:52.549362 20582 leader_election.cc:302] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dda73b9144e4033bde864d0c4d1c54d, ed4a1846e8af4c5ca9d6d07fb51472f8; no voters:
W20260926 21:41:52.549553 20694 tablet.cc:2369] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:52.549691 20717 raft_consensus.cc:2781] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:52.549772 20518 raft_consensus.cc:2454] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4a1846e8af4c5ca9d6d07fb51472f8 in term 1.
I20260926 21:41:52.550657 20717 raft_consensus.cc:686] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 1 LEADER]: Becoming Leader. State: Replica: ed4a1846e8af4c5ca9d6d07fb51472f8, State: Running, Role: LEADER
I20260926 21:41:52.551132 20717 consensus_queue.cc:237] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } }
I20260926 21:41:52.556028 20246 catalog_manager.cc:5495] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed4a1846e8af4c5ca9d6d07fb51472f8 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4a1846e8af4c5ca9d6d07fb51472f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 } health_report { overall_health: UNKNOWN } } }
W20260926 21:41:52.583701 20564 tablet.cc:2369] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:52.608495 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.602488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.0.97.3:44065)
0926 21:41:52.602733 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.602740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.602763 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:41:52.602976 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.602981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.603080 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.603260 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:41:52.603660 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.604464 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.606172 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.606616 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.606773 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.607325 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:41:52.607859 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:41:52.608057 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.608126 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.608216 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 21:41:52.614856 20433 tablet.cc:2369] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:41:52.617319 20518 raft_consensus.cc:1261] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [term 1 FOLLOWER]: Refusing update from remote peer ed4a1846e8af4c5ca9d6d07fb51472f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:52.617311 20387 raft_consensus.cc:1261] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [term 1 FOLLOWER]: Refusing update from remote peer ed4a1846e8af4c5ca9d6d07fb51472f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:41:52.618436 20721 consensus_queue.cc:1035] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dda73b9144e4033bde864d0c4d1c54d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:52.618891 20717 consensus_queue.cc:1035] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0616810c8fab4690a8e47cc5d6b60755" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:41:52.633244 20725 mvcc.cc:205] Tried to move back new op lower bound from 7333719706069065728 to 7333719705813352448. Current Snapshot: MvccSnapshot[applied={T|T < 7333719706069065728}]
I20260926 21:41:52.645156 20246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:52.645484 20246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 21:41:52.649262 20246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc5a2de60c77444584efd5f452c23738: Sending DeleteTablet for 3 replicas of tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7
I20260926 21:41:52.650520 20628 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:52 UTC) from {username='slave'} at 127.0.0.1:43868
I20260926 21:41:52.650615 20498 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:52 UTC) from {username='slave'} at 127.0.0.1:37036
I20260926 21:41:52.651341 20367 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 21:41:52 UTC) from {username='slave'} at 127.0.0.1:33956
I20260926 21:41:52.651484 20734 tablet_replica.cc:330] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: stopping tablet replica
I20260926 21:41:52.652033 20734 raft_consensus.cc:2229] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 21:41:52.652107 20735 tablet_replica.cc:330] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: stopping tablet replica
I20260926 21:41:52.652482 20735 raft_consensus.cc:2229] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:52.652824 20734 raft_consensus.cc:2258] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:52.652979 20735 raft_consensus.cc:2258] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:52.653084 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20305
I20260926 21:41:52.653745 20736 tablet_replica.cc:330] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: stopping tablet replica
I20260926 21:41:52.654135 20736 raft_consensus.cc:2229] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 21:41:52.654618 20736 raft_consensus.cc:2258] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 21:41:52.655059 20734 ts_tablet_manager.cc:1905] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:52.656450 20735 ts_tablet_manager.cc:1905] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:52.658205 20736 ts_tablet_manager.cc:1905] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 21:41:52.662278 20734 ts_tablet_manager.cc:1918] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:52.662458 20734 log.cc:1196] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-2/wal/wals/7f9b5a5223ac4b7da2c40ab386eeb9b7
I20260926 21:41:52.662971 20734 ts_tablet_manager.cc:1939] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P ed4a1846e8af4c5ca9d6d07fb51472f8: Deleting consensus metadata
I20260926 21:41:52.664973 20233 catalog_manager.cc:4841] TS ed4a1846e8af4c5ca9d6d07fb51472f8 (127.0.97.3:44065): tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7 (table test-table [id=74030a5ccbb54fa4bc98b1f677c03c27]) successfully deleted
I20260926 21:41:52.665297 20735 ts_tablet_manager.cc:1918] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:52.665472 20735 log.cc:1196] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-1/wal/wals/7f9b5a5223ac4b7da2c40ab386eeb9b7
I20260926 21:41:52.666026 20735 ts_tablet_manager.cc:1939] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 0616810c8fab4690a8e47cc5d6b60755: Deleting consensus metadata
I20260926 21:41:52.667841 20231 catalog_manager.cc:4841] TS 0616810c8fab4690a8e47cc5d6b60755 (127.0.97.2:33563): tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7 (table test-table [id=74030a5ccbb54fa4bc98b1f677c03c27]) successfully deleted
I20260926 21:41:52.669225 20736 ts_tablet_manager.cc:1918] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 21:41:52.669415 20736 log.cc:1196] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: Deleting WAL directory at /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790458593267894-388-0/minicluster-data/ts-0/wal/wals/7f9b5a5223ac4b7da2c40ab386eeb9b7
I20260926 21:41:52.669934 20736 ts_tablet_manager.cc:1939] T 7f9b5a5223ac4b7da2c40ab386eeb9b7 P 8dda73b9144e4033bde864d0c4d1c54d: Deleting consensus metadata
I20260926 21:41:52.671643 20231 catalog_manager.cc:4841] TS 8dda73b9144e4033bde864d0c4d1c54d (127.0.97.1:37599): tablet 7f9b5a5223ac4b7da2c40ab386eeb9b7 (table test-table [id=74030a5ccbb54fa4bc98b1f677c03c27]) successfully deleted
I20260926 21:41:52.676301 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.655686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.0.97.1:37599)
0926 21:41:52.655934 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.655940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.664719 (+ 8779us) server_negotiation.cc:407] Connection header received
0926 21:41:52.665016 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.665024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.665138 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.665293 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:52.669872 (+ 4579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.671001 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.672944 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.673457 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.673559 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.674793 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:52.674801 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:52.674816 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:52.675388 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:52.675434 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:52.675454 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:52.675552 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.675765 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.675868 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 21:41:52.677093 20427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:52.769989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 20305
I20260926 21:41:52.789036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20436
I20260926 21:41:52.797675 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.789892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.0.97.2:33563)
0926 21:41:52.790091 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.790098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.790119 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:41:52.790330 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.790335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.790445 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.790573 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:52.791012 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.791849 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.793788 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.794433 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.794557 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.795865 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:52.795876 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:52.795896 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:52.796685 (+ 789us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:52.796751 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:52.796778 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:52.796900 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.797058 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.797173 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 21:41:52.798414 20558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:52.894044 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 20436
I20260926 21:41:52.914875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20566
I20260926 21:41:52.922232 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:52.915658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.0.97.3:44065)
0926 21:41:52.915828 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:52.915840 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:52.915862 (+ 22us) server_negotiation.cc:407] Connection header received
0926 21:41:52.916087 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:52.916093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:52.916181 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:52.916299 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:52.916715 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.917609 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.918942 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:52.919409 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:52.919541 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:52.920712 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:52.920722 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:52.920737 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:52.921420 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:52.921478 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:52.921498 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:52.921627 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:52.921767 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 21:41:52.921880 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:41:52.922931 20688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:53.044704 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 20566
I20260926 21:41:53.061285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20215
I20260926 21:41:53.062350 20276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:53.172967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 20215
2026-09-26T21:41:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2889 ms)
[----------] 6 tests from EncryptionPolicyTest (21992 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:41:53 dist-test-slave-lhhq krb5kdc[20744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:41:53 dist-test-slave-lhhq krb5kdc[20744](info): set up 2 sockets
Sep 26 21:41:53 dist-test-slave-lhhq krb5kdc[20744](info): commencing operation
krb5kdc: starting...
W20260926 21:41:54.289705 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:41:54 dist-test-slave-lhhq krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:41:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:41:54.549079 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:54.879798 20757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:54.880141 20757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:54.880326 20757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:54.887490 20757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:54.887647 20757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:54.887743 20757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:54.887857 20757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:54.887925 20757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:54.897534 20757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:54.898874 20757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:54.900223 20757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:54.912122 20763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:54.912381 20762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:54.912741 20765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:54.913010 20757 server_base.cc:1034] running on GCE node
I20260926 21:41:54.913810 20757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:54.915249 20757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:54.916477 20757 hybrid_clock.cc:648] HybridClock initialized: now 1790458914916430 us; error 60 us; skew 500 ppm
I20260926 21:41:54.916824 20757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:54 dist-test-slave-lhhq krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:54.919916 20757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:41:54.921336 20757 webserver.cc:460] Webserver started at http://127.0.0.1:46715/ using document root <none> and password file <none>
I20260926 21:41:54.922026 20757 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:54.922173 20757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:54.922477 20757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:54.925483 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "c02646edfbb24c9abcd5ad8364e6dbde"
format_stamp: "Formatted at 2026-09-26 21:41:54 on dist-test-slave-lhhq"
server_key: "9fe11650c1a1af9a76508242a66ea96a"
server_key_iv: "d34ff07eb9ba4fbc507ca809f0a0c739"
server_key_version: "encryptionkey@0"
I20260926 21:41:54.926254 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "c02646edfbb24c9abcd5ad8364e6dbde"
format_stamp: "Formatted at 2026-09-26 21:41:54 on dist-test-slave-lhhq"
server_key: "9fe11650c1a1af9a76508242a66ea96a"
server_key_iv: "d34ff07eb9ba4fbc507ca809f0a0c739"
server_key_version: "encryptionkey@0"
I20260926 21:41:54.931242 20757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:41:54.934789 20772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:54.936070 20757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:41:54.936249 20757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "c02646edfbb24c9abcd5ad8364e6dbde"
format_stamp: "Formatted at 2026-09-26 21:41:54 on dist-test-slave-lhhq"
server_key: "9fe11650c1a1af9a76508242a66ea96a"
server_key_iv: "d34ff07eb9ba4fbc507ca809f0a0c739"
server_key_version: "encryptionkey@0"
I20260926 21:41:54.936450 20757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:54.948630 20757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:54.952610 20757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:54.952903 20757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:54.971869 20757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39295
I20260926 21:41:54.971916 20823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39295 every 8 connection(s)
I20260926 21:41:54.973315 20757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:54.977177 20824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:54.983582 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 20757
I20260926 21:41:54.983805 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:41:54.984439 388 external_mini_cluster.cc:1428] Setting key b5cb3c7aeb8b85b05c7aa8688c448340
I20260926 21:41:54.989073 20824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde: Bootstrap starting.
I20260926 21:41:54.993150 20824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:54.994493 20824 log.cc:824] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:54.997845 20824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde: No bootstrap required, opened a new log
Sep 26 21:41:54 dist-test-slave-lhhq krb5kdc[20744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:41:55.008745 20824 raft_consensus.cc:348] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02646edfbb24c9abcd5ad8364e6dbde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } }
I20260926 21:41:55.009346 20824 raft_consensus.cc:374] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:55.009438 20824 raft_consensus.cc:729] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02646edfbb24c9abcd5ad8364e6dbde, State: Initialized, Role: FOLLOWER
I20260926 21:41:55.010210 20824 consensus_queue.cc:260] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02646edfbb24c9abcd5ad8364e6dbde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } }
I20260926 21:41:55.010526 20824 raft_consensus.cc:388] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:55.010672 20824 raft_consensus.cc:482] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:55.010860 20824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:55.011956 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:54.986551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37995 (local address 127.0.0.1:39295)
0926 21:41:54.987348 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:54.987363 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:54.987415 (+ 52us) server_negotiation.cc:407] Connection header received
0926 21:41:54.989889 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:54.989918 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:54.990418 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:54.990929 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:54.992182 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:54.994204 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:54.995995 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:54.997020 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:54.997234 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:55.002212 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:55.002224 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:55.002246 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:55.002295 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:55.006441 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:55.007421 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:55.007430 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:55.007440 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:55.007679 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:55.008130 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:55.008135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:55.008138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:55.008888 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:55.009264 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:55.009806 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 21:41:55.010387 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":326,"threads_started":1}
I20260926 21:41:55.013768 20824 raft_consensus.cc:504] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02646edfbb24c9abcd5ad8364e6dbde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } }
I20260926 21:41:55.014339 20824 leader_election.cc:302] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c02646edfbb24c9abcd5ad8364e6dbde; no voters:
I20260926 21:41:55.015064 20824 leader_election.cc:288] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:41:55.015198 20829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:55.016496 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:41:55.016698 20829 raft_consensus.cc:686] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [term 1 LEADER]: Becoming Leader. State: Replica: c02646edfbb24c9abcd5ad8364e6dbde, State: Running, Role: LEADER
I20260926 21:41:55.017280 20829 consensus_queue.cc:237] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02646edfbb24c9abcd5ad8364e6dbde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } }
I20260926 21:41:55.017475 20824 sys_catalog.cc:564] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:55.020819 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [sys.catalog]: SysCatalogTable state changed. Reason: New leader c02646edfbb24c9abcd5ad8364e6dbde. Latest consensus state: current_term: 1 leader_uuid: "c02646edfbb24c9abcd5ad8364e6dbde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02646edfbb24c9abcd5ad8364e6dbde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } } }
I20260926 21:41:55.021136 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:55.022892 20835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:55.022949 20830 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c02646edfbb24c9abcd5ad8364e6dbde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02646edfbb24c9abcd5ad8364e6dbde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39295 } } }
I20260926 21:41:55.023208 20830 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:55.029510 20835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:55.039822 20835 catalog_manager.cc:1347] Generated new cluster ID: bb33e214db2a45a2984b1d462c14a4d7
I20260926 21:41:55.039942 20835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:55.055433 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:55.025793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43075 (local address 127.0.0.1:39295)
0926 21:41:55.026002 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:55.026012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:55.030283 (+ 4271us) server_negotiation.cc:407] Connection header received
0926 21:41:55.030645 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:55.030654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:55.030772 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:55.030884 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:55.033894 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:55.035644 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:55.038467 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:55.039482 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:55.039619 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:55.044424 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:55.044436 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:55.044445 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:41:55.044503 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:55.049616 (+ 5113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:55.051078 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:55.051087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:55.051094 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:55.051335 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:41:55.052109 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:41:55.052117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:41:55.052124 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:41:55.052698 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:55.053089 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:55.054625 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0926 21:41:55.054867 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:41:55.059897 20835 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:41:55.061774 20835 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:55.071048 20835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c02646edfbb24c9abcd5ad8364e6dbde: Generated new TSK 0
I20260926 21:41:55.071903 20835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:41:55.080371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20757
I20260926 21:41:55.082528 20819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:41:55.174559 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 20757
W20260926 21:41:55.191771 20742 connection.cc:535] client connection to 127.0.0.1:39295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39295): Connection reset by peer (error 104)
2026-09-26T21:41:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:41:55 dist-test-slave-lhhq krb5kdc[20856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:41:55 dist-test-slave-lhhq krb5kdc[20856](info): set up 2 sockets
Sep 26 21:41:55 dist-test-slave-lhhq krb5kdc[20856](info): commencing operation
krb5kdc: starting...
W20260926 21:41:56.373494 388 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:41:56 dist-test-slave-lhhq krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:41:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:41:56.632762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:56.964845 20869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:56.965134 20869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:56.965246 20869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:56.972460 20869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:56.972581 20869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:56.972680 20869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:56.972796 20869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:56.972883 20869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:56.982833 20869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:56.984229 20869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:56.985560 20869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:56.997463 20874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:56.997401 20875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:56.997406 20877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:56.998414 20869 server_base.cc:1034] running on GCE node
I20260926 21:41:56.999236 20869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:57.000643 20869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:57.001868 20869 hybrid_clock.cc:648] HybridClock initialized: now 1790458917001830 us; error 55 us; skew 500 ppm
I20260926 21:41:57.002207 20869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:57 dist-test-slave-lhhq krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:57.012416 20869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:41:57.014137 20869 webserver.cc:460] Webserver started at http://127.0.0.1:32999/ using document root <none> and password file <none>
I20260926 21:41:57.014884 20869 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:57.015034 20869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:57.015362 20869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:57.018620 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "3b14f2120758432db098003b78b08f9e"
format_stamp: "Formatted at 2026-09-26 21:41:57 on dist-test-slave-lhhq"
server_key: "aa901ff2481f8a4402ec2f8bfe33c3f3"
server_key_iv: "1cd3e23623ecb978d27bda4127eb8058"
server_key_version: "encryptionkey@0"
I20260926 21:41:57.019397 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "3b14f2120758432db098003b78b08f9e"
format_stamp: "Formatted at 2026-09-26 21:41:57 on dist-test-slave-lhhq"
server_key: "aa901ff2481f8a4402ec2f8bfe33c3f3"
server_key_iv: "1cd3e23623ecb978d27bda4127eb8058"
server_key_version: "encryptionkey@0"
I20260926 21:41:57.024636 20869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:41:57.028116 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:57.029435 20869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 21:41:57.029621 20869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "3b14f2120758432db098003b78b08f9e"
format_stamp: "Formatted at 2026-09-26 21:41:57 on dist-test-slave-lhhq"
server_key: "aa901ff2481f8a4402ec2f8bfe33c3f3"
server_key_iv: "1cd3e23623ecb978d27bda4127eb8058"
server_key_version: "encryptionkey@0"
I20260926 21:41:57.029829 20869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:57.054843 20869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:57.059091 20869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:57.059414 20869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:57.078579 20869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33887
I20260926 21:41:57.078603 20935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33887 every 8 connection(s)
I20260926 21:41:57.080044 20869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:57.084111 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:57.087733 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 20869
I20260926 21:41:57.087932 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:41:57.088370 388 external_mini_cluster.cc:1428] Setting key 80ba35d86235a06e28c605a1d419e9d9
I20260926 21:41:57.096946 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e: Bootstrap starting.
I20260926 21:41:57.100837 20936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:57.102095 20936 log.cc:824] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:57.102170 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:57.090257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50131 (local address 127.0.0.1:33887)
0926 21:41:57.091056 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:57.091068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:57.091109 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:41:57.092816 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:57.092839 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:57.093348 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:57.093705 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:57.094320 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:57.095619 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:57.097518 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:57.098180 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:57.098319 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:57.099767 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:57.099775 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:57.099788 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:57.100324 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:57.100398 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:57.100423 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:57.100542 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:57.100928 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 21:41:57.101125 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":357,"threads_started":1}
I20260926 21:41:57.105268 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e: No bootstrap required, opened a new log
I20260926 21:41:57.106304 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:41:57.116312 20936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b14f2120758432db098003b78b08f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33887 } }
I20260926 21:41:57.116865 20936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:57.116979 20936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b14f2120758432db098003b78b08f9e, State: Initialized, Role: FOLLOWER
I20260926 21:41:57.117808 20936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b14f2120758432db098003b78b08f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33887 } }
I20260926 21:41:57.118134 20936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:57.118279 20936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:57.118489 20936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:57.118834 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:57.111556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48631 (local address 127.0.0.1:33887)
0926 21:41:57.111808 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:57.111815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:57.112212 (+ 397us) server_negotiation.cc:407] Connection header received
0926 21:41:57.112409 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:57.112412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:57.112476 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:57.112559 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:57.113176 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:57.113992 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:57.115923 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:41:57.116351 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:41:57.116494 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:41:57.117971 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:57.117976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:57.117979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:57.117995 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:57.118082 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:57.118107 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:57.118223 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:57.118461 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 21:41:57.118520 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:41:57.121654 20936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b14f2120758432db098003b78b08f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33887 } }
I20260926 21:41:57.122166 20936 leader_election.cc:302] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b14f2120758432db098003b78b08f9e; no voters:
I20260926 21:41:57.122787 20936 leader_election.cc:288] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:41:57.122956 20946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:57.123788 20946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [term 1 LEADER]: Becoming Leader. State: Replica: 3b14f2120758432db098003b78b08f9e, State: Running, Role: LEADER
I20260926 21:41:57.124337 20946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b14f2120758432db098003b78b08f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33887 } }
I20260926 21:41:57.125237 20936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:57.134878 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b14f2120758432db098003b78b08f9e. Latest consensus state: current_term: 1 leader_uuid: "3b14f2120758432db098003b78b08f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b14f2120758432db098003b78b08f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33887 } } }
I20260926 21:41:57.134793 20947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b14f2120758432db098003b78b08f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b14f2120758432db098003b78b08f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33887 } } }
I20260926 21:41:57.135363 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:57.135368 20947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e [sys.catalog]: This master's current role is: LEADER
W20260926 21:41:57.141790 20961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:41:57.141929 20961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:41:57.142083 20962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:57.144352 20962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:57.153878 20962 catalog_manager.cc:1347] Generated new cluster ID: ab9921e707234883809b2ba6f4dd6b49
I20260926 21:41:57.154007 20962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:57.178735 20962 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:41:57.180653 20962 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:57.188329 20962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b14f2120758432db098003b78b08f9e: Generated new TSK 0
I20260926 21:41:57.189164 20962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:41:57.195525 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20869
W20260926 21:41:57.196727 20931 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50131
W20260926 21:41:57.197559 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20869: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:41:57.197751 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 20869
W20260926 21:41:57.211959 20854 connection.cc:535] client connection to 127.0.0.1:33887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33887): Connection reset by peer (error 104)
2026-09-26T21:41:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:41:57 dist-test-slave-lhhq krb5kdc[20967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:41:57 dist-test-slave-lhhq krb5kdc[20967](info): set up 2 sockets
Sep 26 21:41:57 dist-test-slave-lhhq krb5kdc[20967](info): commencing operation
krb5kdc: starting...
W20260926 21:41:58.305634 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:41:58 dist-test-slave-lhhq krb5kdc[20967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:41:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:41:58.560907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:41:58.892133 20980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:41:58.892518 20980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:41:58.892685 20980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:41:58.900002 20980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:41:58.900179 20980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:41:58.900295 20980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:41:58.900408 20980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:41:58.900543 20980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:41:58.910835 20980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:41:58.912216 20980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:41:58.913582 20980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:41:58.926090 20986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:58.926072 20988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:41:58.926167 20985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:41:58.931191 20980 server_base.cc:1034] running on GCE node
I20260926 21:41:58.932109 20980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:41:58.933782 20980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:41:58.935025 20980 hybrid_clock.cc:648] HybridClock initialized: now 1790458918934978 us; error 81 us; skew 500 ppm
I20260926 21:41:58.935403 20980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:41:58 dist-test-slave-lhhq krb5kdc[20967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:41:58.938761 20980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:41:58.940441 20980 webserver.cc:460] Webserver started at http://127.0.0.1:39675/ using document root <none> and password file <none>
I20260926 21:41:58.941270 20980 fs_manager.cc:359] Metadata directory not provided
I20260926 21:41:58.941439 20980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:41:58.941774 20980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:41:58.944939 20980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "da430272a75f4277b28e47d7968cbf96"
format_stamp: "Formatted at 2026-09-26 21:41:58 on dist-test-slave-lhhq"
server_key: "cc6f017978a65dd09568ae31408025a0"
server_key_iv: "0592efeab6c4ab90c458cad6a0734752"
server_key_version: "encryptionkey@0"
I20260926 21:41:58.945777 20980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "da430272a75f4277b28e47d7968cbf96"
format_stamp: "Formatted at 2026-09-26 21:41:58 on dist-test-slave-lhhq"
server_key: "cc6f017978a65dd09568ae31408025a0"
server_key_iv: "0592efeab6c4ab90c458cad6a0734752"
server_key_version: "encryptionkey@0"
I20260926 21:41:58.951159 20980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 21:41:58.954697 20995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:41:58.956038 20980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:41:58.956233 20980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "da430272a75f4277b28e47d7968cbf96"
format_stamp: "Formatted at 2026-09-26 21:41:58 on dist-test-slave-lhhq"
server_key: "cc6f017978a65dd09568ae31408025a0"
server_key_iv: "0592efeab6c4ab90c458cad6a0734752"
server_key_version: "encryptionkey@0"
I20260926 21:41:58.956449 20980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:41:58.962252 20980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:41:58.966532 20980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:41:58.966847 20980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:41:58.986330 20980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45973
I20260926 21:41:58.986353 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45973 every 8 connection(s)
I20260926 21:41:58.987855 20980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:41:58.991717 21047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:41:58.995409 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 20980
I20260926 21:41:58.995612 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:41:58.996063 388 external_mini_cluster.cc:1428] Setting key e6452b53528c77fabf42841b6aaa0f8a
I20260926 21:41:59.004396 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:58.997987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46581 (local address 127.0.0.1:45973)
0926 21:41:58.998845 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:58.998859 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:58.998903 (+ 44us) server_negotiation.cc:407] Connection header received
0926 21:41:59.000683 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:59.000709 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:59.001104 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:59.001635 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:59.002750 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:59.002756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:59.002770 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:59.002784 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:59.002841 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:59.002864 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:59.003097 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:59.003241 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 21:41:59.003441 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":393,"threads_started":1}
I20260926 21:41:59.004781 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96: Bootstrap starting.
I20260926 21:41:59.008306 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:41:59.008520 21047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96: Neither blocks nor log segments found. Creating new log.
I20260926 21:41:59.009621 21047 log.cc:824] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96: Log is configured to *not* fsync() on all Append() calls
I20260926 21:41:59.012921 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96: No bootstrap required, opened a new log
I20260926 21:41:59.015378 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:41:59.012787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41637 (local address 127.0.0.1:45973)
0926 21:41:59.012984 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:41:59.012991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:41:59.013382 (+ 391us) server_negotiation.cc:407] Connection header received
0926 21:41:59.013598 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:41:59.013601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:41:59.013647 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:41:59.013753 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:41:59.014714 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:41:59.014718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:41:59.014724 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:41:59.014739 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:41:59.014776 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:41:59.014804 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:41:59.014946 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 21:41:59.015046 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 21:41:59.015104 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 21:41:59.021512 21047 raft_consensus.cc:348] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da430272a75f4277b28e47d7968cbf96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260926 21:41:59.021955 21047 raft_consensus.cc:374] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:41:59.022076 21047 raft_consensus.cc:729] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da430272a75f4277b28e47d7968cbf96, State: Initialized, Role: FOLLOWER
I20260926 21:41:59.022639 21047 consensus_queue.cc:260] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da430272a75f4277b28e47d7968cbf96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260926 21:41:59.022890 21047 raft_consensus.cc:388] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:41:59.023036 21047 raft_consensus.cc:482] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:41:59.023211 21047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:41:59.025955 21047 raft_consensus.cc:504] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da430272a75f4277b28e47d7968cbf96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260926 21:41:59.026460 21047 leader_election.cc:302] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da430272a75f4277b28e47d7968cbf96; no voters:
I20260926 21:41:59.027036 21047 leader_election.cc:288] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:41:59.027192 21057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:41:59.027980 21057 raft_consensus.cc:686] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [term 1 LEADER]: Becoming Leader. State: Replica: da430272a75f4277b28e47d7968cbf96, State: Running, Role: LEADER
I20260926 21:41:59.028510 21057 consensus_queue.cc:237] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da430272a75f4277b28e47d7968cbf96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260926 21:41:59.029277 21047 sys_catalog.cc:564] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:41:59.031589 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da430272a75f4277b28e47d7968cbf96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da430272a75f4277b28e47d7968cbf96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } } }
I20260926 21:41:59.031627 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da430272a75f4277b28e47d7968cbf96. Latest consensus state: current_term: 1 leader_uuid: "da430272a75f4277b28e47d7968cbf96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da430272a75f4277b28e47d7968cbf96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } } }
I20260926 21:41:59.031976 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:59.032012 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96 [sys.catalog]: This master's current role is: LEADER
I20260926 21:41:59.035203 21062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:41:59.040761 21062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:41:59.050091 21062 catalog_manager.cc:1347] Generated new cluster ID: 6fcb7f52240d4b168820fc1279c589a5
I20260926 21:41:59.050343 21062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:41:59.094024 21062 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:41:59.094585 21062 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:41:59.107604 21062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da430272a75f4277b28e47d7968cbf96: Generated new TSK 0
I20260926 21:41:59.108558 21062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:41:59.124951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20980
W20260926 21:41:59.126824 21042 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46581
W20260926 21:41:59.127561 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 20980: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:41:59.127723 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 20980
2026-09-26T21:41:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:41:59 dist-test-slave-lhhq krb5kdc[21078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:41:59 dist-test-slave-lhhq krb5kdc[21078](info): set up 2 sockets
Sep 26 21:41:59 dist-test-slave-lhhq krb5kdc[21078](info): commencing operation
krb5kdc: starting...
W20260926 21:42:00.245533 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:00 dist-test-slave-lhhq krb5kdc[21078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:42:00.504992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:00.838799 21091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:00.839102 21091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:00.839305 21091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:00.846753 21091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:00.846860 21091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:00.846974 21091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:00.847059 21091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:00.847139 21091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:00.857385 21091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:00.858707 21091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:00.860059 21091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:00.872061 21096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:00.872133 21097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:00.872107 21099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:00.873157 21091 server_base.cc:1034] running on GCE node
I20260926 21:42:00.874006 21091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:00.875427 21091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:00.876657 21091 hybrid_clock.cc:648] HybridClock initialized: now 1790458920876614 us; error 38 us; skew 500 ppm
I20260926 21:42:00.877008 21091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:00 dist-test-slave-lhhq krb5kdc[21078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:00.880187 21091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:42:00.881829 21091 webserver.cc:460] Webserver started at http://127.0.0.1:32883/ using document root <none> and password file <none>
I20260926 21:42:00.882526 21091 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:00.882674 21091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:00.882989 21091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:00.886075 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9"
format_stamp: "Formatted at 2026-09-26 21:42:00 on dist-test-slave-lhhq"
server_key: "709b487e06fcef273a9211d0004a28f2"
server_key_iv: "e006243167c483b4faf3ef95a101724e"
server_key_version: "encryptionkey@0"
I20260926 21:42:00.886871 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9"
format_stamp: "Formatted at 2026-09-26 21:42:00 on dist-test-slave-lhhq"
server_key: "709b487e06fcef273a9211d0004a28f2"
server_key_iv: "e006243167c483b4faf3ef95a101724e"
server_key_version: "encryptionkey@0"
I20260926 21:42:00.892055 21091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:42:00.895635 21106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:00.896943 21091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:42:00.897176 21091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9"
format_stamp: "Formatted at 2026-09-26 21:42:00 on dist-test-slave-lhhq"
server_key: "709b487e06fcef273a9211d0004a28f2"
server_key_iv: "e006243167c483b4faf3ef95a101724e"
server_key_version: "encryptionkey@0"
I20260926 21:42:00.897449 21091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:00.912089 21091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:00.916567 21091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:00.916864 21091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:00.935969 21091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41387
I20260926 21:42:00.936014 21157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41387 every 8 connection(s)
I20260926 21:42:00.937422 21091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:00.939251 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 21091
I20260926 21:42:00.939456 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:00.939913 388 external_mini_cluster.cc:1428] Setting key 5ab162542cd6c50d10b83bfa2a6002d8
I20260926 21:42:00.941815 21158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:00.955399 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9: Bootstrap starting.
Sep 26 21:42:00 dist-test-slave-lhhq krb5kdc[21078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:42:00.959504 21158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:00.960625 21158 log.cc:824] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:00.963980 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9: No bootstrap required, opened a new log
I20260926 21:42:00.967620 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:00.941657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45957 (local address 127.0.0.1:41387)
0926 21:42:00.945781 (+ 4124us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:00.945794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:00.945840 (+ 46us) server_negotiation.cc:407] Connection header received
0926 21:42:00.947497 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:00.947523 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:00.947940 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:00.948287 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:00.950092 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:00.951389 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:00.953432 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:00.954090 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:00.954222 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:00.959365 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:00.959373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:00.959391 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:00.959428 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:00.963190 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:00.963995 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:00.964005 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:00.964013 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:00.964210 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:00.964639 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:00.964647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:00.964652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:00.965401 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:00.965720 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:00.966195 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 21:42:00.966617 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3897,"thread_start_us":3669,"threads_started":1}
I20260926 21:42:00.971339 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:00.972410 21158 raft_consensus.cc:348] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41387 } }
I20260926 21:42:00.972757 21158 raft_consensus.cc:374] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:00.972854 21158 raft_consensus.cc:729] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafaa207dc124dbbb8fd6f6a4e39adf9, State: Initialized, Role: FOLLOWER
I20260926 21:42:00.973470 21158 consensus_queue.cc:260] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41387 } }
I20260926 21:42:00.973691 21158 raft_consensus.cc:388] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:00.973841 21158 raft_consensus.cc:482] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:00.974042 21158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:00.976769 21158 raft_consensus.cc:504] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41387 } }
I20260926 21:42:00.977335 21158 leader_election.cc:302] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eafaa207dc124dbbb8fd6f6a4e39adf9; no voters:
I20260926 21:42:00.977989 21158 leader_election.cc:288] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:00.978088 21168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:00.980619 21158 sys_catalog.cc:564] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:00.984735 21168 raft_consensus.cc:686] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [term 1 LEADER]: Becoming Leader. State: Replica: eafaa207dc124dbbb8fd6f6a4e39adf9, State: Running, Role: LEADER
I20260926 21:42:00.985517 21168 consensus_queue.cc:237] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41387 } }
I20260926 21:42:00.989415 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41387 } } }
I20260926 21:42:00.989830 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:00.989118 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eafaa207dc124dbbb8fd6f6a4e39adf9. Latest consensus state: current_term: 1 leader_uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafaa207dc124dbbb8fd6f6a4e39adf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41387 } } }
I20260926 21:42:00.990468 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:00.992563 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:00.975923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59243 (local address 127.0.0.1:41387)
0926 21:42:00.976098 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:00.976106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:00.976462 (+ 356us) server_negotiation.cc:407] Connection header received
0926 21:42:00.976650 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:00.976655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:00.976729 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:00.976844 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:00.978532 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:00.979462 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:00.980844 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:00.981361 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:00.981509 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:00.984256 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:00.984268 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:00.984272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:00.984306 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:00.987905 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:00.988896 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:00.988904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:00.988908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:00.989079 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:00.989598 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:00.989609 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:00.989612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:00.990154 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:00.991480 (+ 1326us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:00.991711 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 21:42:00.992035 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:42:00.994239 21180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:01.000141 21180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:01.010337 21180 catalog_manager.cc:1347] Generated new cluster ID: d5c1c1729ff64c4aa17b5e689902e0df
I20260926 21:42:01.010462 21180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:01.026575 21180 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:01.028488 21180 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:01.040103 21180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eafaa207dc124dbbb8fd6f6a4e39adf9: Generated new TSK 0
I20260926 21:42:01.040977 21180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:01.053550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21091
I20260926 21:42:01.054983 21153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:01.146158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 21091
2026-09-26T21:42:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2017 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:01 dist-test-slave-lhhq krb5kdc[21190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:01 dist-test-slave-lhhq krb5kdc[21190](info): set up 2 sockets
Sep 26 21:42:01 dist-test-slave-lhhq krb5kdc[21190](info): commencing operation
krb5kdc: starting...
W20260926 21:42:02.249588 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:02 dist-test-slave-lhhq krb5kdc[21190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:42:02.516499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:02.848165 21203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:02.848493 21203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:02.848590 21203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:02.856127 21203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:02.856251 21203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:02.856299 21203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:02.856374 21203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:02.856419 21203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:02.866473 21203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:02.867771 21203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:02.869050 21203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:02.881896 21208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:02.881954 21209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:02.881978 21211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:02.883359 21203 server_base.cc:1034] running on GCE node
I20260926 21:42:02.884195 21203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:02.885701 21203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:02.886946 21203 hybrid_clock.cc:648] HybridClock initialized: now 1790458922886899 us; error 40 us; skew 500 ppm
I20260926 21:42:02.887301 21203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:02 dist-test-slave-lhhq krb5kdc[21190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:02.905812 21203 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:42:02.907498 21203 webserver.cc:460] Webserver started at http://127.0.0.1:44085/ using document root <none> and password file <none>
I20260926 21:42:02.908243 21203 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:02.908396 21203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:02.908727 21203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:02.911948 21203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "1b3e277492f042f88a2975b3f6bb9c96"
format_stamp: "Formatted at 2026-09-26 21:42:02 on dist-test-slave-lhhq"
server_key: "08cd28d7d321aa0597c355d29658c5b4"
server_key_iv: "6958dd2a44e45ad5e6b883775ec4c6db"
server_key_version: "encryptionkey@0"
I20260926 21:42:02.912748 21203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "1b3e277492f042f88a2975b3f6bb9c96"
format_stamp: "Formatted at 2026-09-26 21:42:02 on dist-test-slave-lhhq"
server_key: "08cd28d7d321aa0597c355d29658c5b4"
server_key_iv: "6958dd2a44e45ad5e6b883775ec4c6db"
server_key_version: "encryptionkey@0"
I20260926 21:42:02.918318 21203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:42:02.922026 21218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:02.923557 21203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:42:02.923712 21203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "1b3e277492f042f88a2975b3f6bb9c96"
format_stamp: "Formatted at 2026-09-26 21:42:02 on dist-test-slave-lhhq"
server_key: "08cd28d7d321aa0597c355d29658c5b4"
server_key_iv: "6958dd2a44e45ad5e6b883775ec4c6db"
server_key_version: "encryptionkey@0"
I20260926 21:42:02.923902 21203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:02.938910 21203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:02.943470 21203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:02.943708 21203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:02.963114 21203 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32927
I20260926 21:42:02.963143 21269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32927 every 8 connection(s)
I20260926 21:42:02.964512 21203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:02.968366 21270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:02.972659 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 21203
I20260926 21:42:02.972884 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:02.973363 388 external_mini_cluster.cc:1428] Setting key 22e702fdf90b802fbde97ff8bc72ef9e
I20260926 21:42:02.979980 21270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96: Bootstrap starting.
I20260926 21:42:02.983970 21270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:02.985288 21270 log.cc:824] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:02.988790 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:02.975160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59493 (local address 127.0.0.1:32927)
0926 21:42:02.975951 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:02.975964 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:02.976008 (+ 44us) server_negotiation.cc:407] Connection header received
0926 21:42:02.978240 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:02.978271 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:02.978797 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:02.979283 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:02.980458 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:02.981876 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:02.983802 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:02.984581 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:02.984717 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:02.986141 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:02.986157 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:02.986178 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:02.987028 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:02.987106 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:02.987137 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:02.987228 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:02.987485 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 21:42:02.987790 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":367,"threads_started":1}
I20260926 21:42:02.988811 21270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96: No bootstrap required, opened a new log
I20260926 21:42:02.992822 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:02.997572 21270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3e277492f042f88a2975b3f6bb9c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260926 21:42:02.998123 21270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:02.998265 21270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3e277492f042f88a2975b3f6bb9c96, State: Initialized, Role: FOLLOWER
I20260926 21:42:02.999040 21270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3e277492f042f88a2975b3f6bb9c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260926 21:42:02.999317 21270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:02.999482 21270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:02.999677 21270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:03.002904 21270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3e277492f042f88a2975b3f6bb9c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260926 21:42:03.003612 21270 leader_election.cc:302] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b3e277492f042f88a2975b3f6bb9c96; no voters:
I20260926 21:42:03.004410 21270 leader_election.cc:288] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:03.004470 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:02.997330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34939 (local address 127.0.0.1:32927)
0926 21:42:02.997539 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:02.997545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:02.997866 (+ 321us) server_negotiation.cc:407] Connection header received
0926 21:42:02.998083 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:02.998088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:02.998164 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:02.998297 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:02.998974 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:02.999953 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:03.001855 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:03.002302 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:03.002400 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:03.003659 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:03.003664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:03.003667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:03.003685 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:03.003725 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:03.003740 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:03.003851 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:03.004004 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 21:42:03.004096 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:42:03.004520 21280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:03.005326 21280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3e277492f042f88a2975b3f6bb9c96, State: Running, Role: LEADER
I20260926 21:42:03.006208 21280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3e277492f042f88a2975b3f6bb9c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260926 21:42:03.008343 21270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:03.009496 21282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b3e277492f042f88a2975b3f6bb9c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3e277492f042f88a2975b3f6bb9c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } } }
I20260926 21:42:03.009552 21281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b3e277492f042f88a2975b3f6bb9c96. Latest consensus state: current_term: 1 leader_uuid: "1b3e277492f042f88a2975b3f6bb9c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3e277492f042f88a2975b3f6bb9c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } } }
I20260926 21:42:03.010252 21281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:03.011801 21282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:03.012504 21285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:03.019548 21285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:03.029394 21285 catalog_manager.cc:1347] Generated new cluster ID: 283a76cb35e14845a6c2dfa8b6566f7b
I20260926 21:42:03.029527 21285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:03.038692 21285 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:03.040051 21285 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:03.046857 21285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b3e277492f042f88a2975b3f6bb9c96: Generated new TSK 0
I20260926 21:42:03.047715 21285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:03.058468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21203
W20260926 21:42:03.059581 21265 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59493
W20260926 21:42:03.060284 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:03.060400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 21203
2026-09-26T21:42:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:03 dist-test-slave-lhhq krb5kdc[21301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:03 dist-test-slave-lhhq krb5kdc[21301](info): set up 2 sockets
Sep 26 21:42:03 dist-test-slave-lhhq krb5kdc[21301](info): commencing operation
krb5kdc: starting...
W20260926 21:42:04.173506 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:04 dist-test-slave-lhhq krb5kdc[21301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:42:04.428099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:04.758198 21314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:04.758502 21314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:04.758698 21314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:04.766021 21314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:04.766146 21314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:04.766266 21314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:04.766378 21314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:04.766465 21314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:04.776696 21314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:04.778102 21314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:04.779629 21314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:04.791620 21320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:04.791735 21319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:04.791735 21322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:04.792982 21314 server_base.cc:1034] running on GCE node
I20260926 21:42:04.793814 21314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:04.795254 21314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:04.796464 21314 hybrid_clock.cc:648] HybridClock initialized: now 1790458924796425 us; error 29 us; skew 500 ppm
I20260926 21:42:04.796801 21314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:04 dist-test-slave-lhhq krb5kdc[21301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:04.799955 21314 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:42:04.801608 21314 webserver.cc:460] Webserver started at http://127.0.0.1:33819/ using document root <none> and password file <none>
I20260926 21:42:04.802337 21314 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:04.802482 21314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:04.802794 21314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:04.805919 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "85e2c0b2f5f44f31a0d10f18e569a0da"
format_stamp: "Formatted at 2026-09-26 21:42:04 on dist-test-slave-lhhq"
server_key: "99838f047a5c327dceb473cffe65fa88"
server_key_iv: "b8b2bde29e3f429865a652831861ebd8"
server_key_version: "encryptionkey@0"
I20260926 21:42:04.806700 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "85e2c0b2f5f44f31a0d10f18e569a0da"
format_stamp: "Formatted at 2026-09-26 21:42:04 on dist-test-slave-lhhq"
server_key: "99838f047a5c327dceb473cffe65fa88"
server_key_iv: "b8b2bde29e3f429865a652831861ebd8"
server_key_version: "encryptionkey@0"
I20260926 21:42:04.812062 21314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 21:42:04.815677 21329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:04.817133 21314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:42:04.817379 21314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "85e2c0b2f5f44f31a0d10f18e569a0da"
format_stamp: "Formatted at 2026-09-26 21:42:04 on dist-test-slave-lhhq"
server_key: "99838f047a5c327dceb473cffe65fa88"
server_key_iv: "b8b2bde29e3f429865a652831861ebd8"
server_key_version: "encryptionkey@0"
I20260926 21:42:04.817595 21314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:04.823462 21314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:04.827725 21314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:04.828034 21314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:04.847435 21314 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42853
I20260926 21:42:04.847484 21380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42853 every 8 connection(s)
I20260926 21:42:04.848893 21314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:04.852237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 21314
I20260926 21:42:04.852437 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:04.852870 388 external_mini_cluster.cc:1428] Setting key b3a9a52e50761857e49e59e5d44fd0a2
I20260926 21:42:04.852916 21381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:04.861245 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:04.854755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48863 (local address 127.0.0.1:42853)
0926 21:42:04.855593 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:04.855604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:04.855649 (+ 45us) server_negotiation.cc:407] Connection header received
0926 21:42:04.857724 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:04.857748 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:04.858103 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:04.858450 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:04.859616 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:04.859623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:04.859639 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:04.859653 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:04.859718 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:04.859742 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:04.859921 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:04.860030 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 21:42:04.860246 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":370,"threads_started":1}
I20260926 21:42:04.864789 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:04.866726 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da: Bootstrap starting.
I20260926 21:42:04.870103 21381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:04.871186 21381 log.cc:824] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:04.872056 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:04.869247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56525 (local address 127.0.0.1:42853)
0926 21:42:04.869488 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:04.869498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:04.869998 (+ 500us) server_negotiation.cc:407] Connection header received
0926 21:42:04.870197 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:04.870202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:04.870268 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:04.870358 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:04.871289 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:04.871295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:04.871300 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:04.871317 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:04.871365 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:04.871384 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:04.871513 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:04.871605 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 21:42:04.871669 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:42:04.874238 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da: No bootstrap required, opened a new log
I20260926 21:42:04.881302 21381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e2c0b2f5f44f31a0d10f18e569a0da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } }
I20260926 21:42:04.881668 21381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:04.881743 21381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e2c0b2f5f44f31a0d10f18e569a0da, State: Initialized, Role: FOLLOWER
I20260926 21:42:04.882233 21381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e2c0b2f5f44f31a0d10f18e569a0da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } }
I20260926 21:42:04.882434 21381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:04.882530 21381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:04.882651 21381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:04.884971 21381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e2c0b2f5f44f31a0d10f18e569a0da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } }
I20260926 21:42:04.885502 21381 leader_election.cc:302] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85e2c0b2f5f44f31a0d10f18e569a0da; no voters:
I20260926 21:42:04.886036 21381 leader_election.cc:288] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:04.886178 21391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:04.886848 21391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [term 1 LEADER]: Becoming Leader. State: Replica: 85e2c0b2f5f44f31a0d10f18e569a0da, State: Running, Role: LEADER
I20260926 21:42:04.887320 21391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e2c0b2f5f44f31a0d10f18e569a0da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } }
I20260926 21:42:04.888103 21381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:04.889902 21392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85e2c0b2f5f44f31a0d10f18e569a0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e2c0b2f5f44f31a0d10f18e569a0da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } } }
I20260926 21:42:04.890379 21392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:04.890241 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e2c0b2f5f44f31a0d10f18e569a0da. Latest consensus state: current_term: 1 leader_uuid: "85e2c0b2f5f44f31a0d10f18e569a0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e2c0b2f5f44f31a0d10f18e569a0da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } } }
I20260926 21:42:04.890496 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:04.891193 21396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:04.896320 21396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:04.905644 21396 catalog_manager.cc:1347] Generated new cluster ID: 71b4df31a4b140d08d3cc614c93718ec
I20260926 21:42:04.905836 21396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:04.918762 21396 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:04.919191 21396 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:04.925127 21396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85e2c0b2f5f44f31a0d10f18e569a0da: Generated new TSK 0
I20260926 21:42:04.925997 21396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:04.940274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21314
W20260926 21:42:04.941368 21376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48863
W20260926 21:42:04.942209 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21314: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:04.942375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 21314
2026-09-26T21:42:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:05 dist-test-slave-lhhq krb5kdc[21412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:05 dist-test-slave-lhhq krb5kdc[21412](info): set up 2 sockets
Sep 26 21:42:05 dist-test-slave-lhhq krb5kdc[21412](info): commencing operation
krb5kdc: starting...
W20260926 21:42:06.045449 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:06 dist-test-slave-lhhq krb5kdc[21412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:42:06.294137 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:06.623041 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:06.623432 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:06.623600 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:06.630944 21425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:06.631107 21425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:06.631222 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:06.631337 21425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:06.631441 21425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:06.641258 21425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:06.642735 21425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:06.644086 21425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:06.655722 21430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:06.655817 21431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:06.655859 21433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:06.656723 21425 server_base.cc:1034] running on GCE node
I20260926 21:42:06.657549 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:06.659029 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:06.660248 21425 hybrid_clock.cc:648] HybridClock initialized: now 1790458926660203 us; error 48 us; skew 500 ppm
I20260926 21:42:06.660586 21425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:06 dist-test-slave-lhhq krb5kdc[21412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:06.663962 21425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:42:06.666244 21425 webserver.cc:460] Webserver started at http://127.0.0.1:44917/ using document root <none> and password file <none>
I20260926 21:42:06.666927 21425 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:06.667075 21425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:06.667377 21425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:06.670292 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "4fc7cabed3bd465ca24a131d100109fc"
format_stamp: "Formatted at 2026-09-26 21:42:06 on dist-test-slave-lhhq"
server_key: "4bfc79491fcb88360ae4323e2a1c4ec6"
server_key_iv: "d80ab97f5c1b78036b4fe4f7efdfabbf"
server_key_version: "encryptionkey@0"
I20260926 21:42:06.671039 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "4fc7cabed3bd465ca24a131d100109fc"
format_stamp: "Formatted at 2026-09-26 21:42:06 on dist-test-slave-lhhq"
server_key: "4bfc79491fcb88360ae4323e2a1c4ec6"
server_key_iv: "d80ab97f5c1b78036b4fe4f7efdfabbf"
server_key_version: "encryptionkey@0"
I20260926 21:42:06.676077 21425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:42:06.679474 21440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:06.680810 21425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:42:06.681016 21425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "4fc7cabed3bd465ca24a131d100109fc"
format_stamp: "Formatted at 2026-09-26 21:42:06 on dist-test-slave-lhhq"
server_key: "4bfc79491fcb88360ae4323e2a1c4ec6"
server_key_iv: "d80ab97f5c1b78036b4fe4f7efdfabbf"
server_key_version: "encryptionkey@0"
I20260926 21:42:06.681277 21425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:06.702913 21425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:06.707372 21425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:06.707661 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:06.726480 21425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45939
I20260926 21:42:06.726512 21491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45939 every 8 connection(s)
I20260926 21:42:06.727910 21425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:06.728989 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 21425
I20260926 21:42:06.729212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:06.729784 388 external_mini_cluster.cc:1428] Setting key 61d6536335e1a21c20ce1814003664ec
I20260926 21:42:06.732592 21492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 21:42:06 dist-test-slave-lhhq krb5kdc[21412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:42:06.743485 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc: Bootstrap starting.
I20260926 21:42:06.746940 21492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:06.748050 21492 log.cc:824] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:06.751036 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc: No bootstrap required, opened a new log
I20260926 21:42:06.752224 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:06.731623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:38811 (local address 127.0.0.1:45939)
0926 21:42:06.732342 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:06.732354 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:06.732395 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:42:06.734167 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:06.734190 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:06.734573 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:06.734983 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:06.736436 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:06.737845 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:06.739142 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:06.739838 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:06.740055 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:06.744634 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:06.744642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:06.744658 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:06.744704 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:06.747990 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:06.748875 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:06.748883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:06.748889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:06.749059 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:06.749586 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:06.749595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:06.749597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:06.750103 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:06.750444 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:06.750904 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 21:42:06.751249 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":311,"threads_started":1}
I20260926 21:42:06.756158 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:06.759261 21492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc7cabed3bd465ca24a131d100109fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } }
I20260926 21:42:06.759644 21492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:06.759756 21492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc7cabed3bd465ca24a131d100109fc, State: Initialized, Role: FOLLOWER
I20260926 21:42:06.760301 21492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc7cabed3bd465ca24a131d100109fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } }
I20260926 21:42:06.760635 21492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:06.760807 21492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:06.760991 21492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:06.764281 21492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc7cabed3bd465ca24a131d100109fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } }
I20260926 21:42:06.764918 21492 leader_election.cc:302] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc7cabed3bd465ca24a131d100109fc; no voters:
I20260926 21:42:06.765620 21492 leader_election.cc:288] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:06.765748 21502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:06.767030 21502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [term 1 LEADER]: Becoming Leader. State: Replica: 4fc7cabed3bd465ca24a131d100109fc, State: Running, Role: LEADER
I20260926 21:42:06.767704 21502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc7cabed3bd465ca24a131d100109fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } }
I20260926 21:42:06.768080 21492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:06.778625 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fc7cabed3bd465ca24a131d100109fc. Latest consensus state: current_term: 1 leader_uuid: "4fc7cabed3bd465ca24a131d100109fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc7cabed3bd465ca24a131d100109fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } } }
I20260926 21:42:06.778900 21503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fc7cabed3bd465ca24a131d100109fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc7cabed3bd465ca24a131d100109fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } } }
I20260926 21:42:06.780011 21503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:06.780280 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:06.781422 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:06.760744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:34551 (local address 127.0.0.1:45939)
0926 21:42:06.760923 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:06.760929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:06.761534 (+ 605us) server_negotiation.cc:407] Connection header received
0926 21:42:06.761726 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:06.761730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:06.761799 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:06.761882 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:06.763616 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:06.764487 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:06.766465 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:06.767143 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:06.767263 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:06.770913 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:06.770922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:06.770930 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:06.770985 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:06.775750 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:06.776908 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:06.776913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:06.776916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:06.777137 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:06.777870 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:06.777878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:06.777882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:06.778395 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:06.779746 (+ 1351us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:06.780609 (+ 863us) server_negotiation.cc:299] Negotiation successful
0926 21:42:06.780880 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 21:42:06.790473 21518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:06.796726 21518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:06.805698 21518 catalog_manager.cc:1347] Generated new cluster ID: aba11b4615c24fbd8c45c7534743ef02
I20260926 21:42:06.805828 21518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:06.827569 21518 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:06.829527 21518 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:06.841029 21518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fc7cabed3bd465ca24a131d100109fc: Generated new TSK 0
I20260926 21:42:06.841907 21518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:06.855257 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21425
I20260926 21:42:06.856412 21487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:06.950428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 21425
W20260926 21:42:06.967689 21410 connection.cc:535] client connection to 127.0.0.1:45939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45939): Connection reset by peer (error 104)
2026-09-26T21:42:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2008 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:07 dist-test-slave-lhhq krb5kdc[21525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:07 dist-test-slave-lhhq krb5kdc[21525](info): set up 2 sockets
Sep 26 21:42:07 dist-test-slave-lhhq krb5kdc[21525](info): commencing operation
krb5kdc: starting...
W20260926 21:42:08.049759 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:08 dist-test-slave-lhhq krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:42:08.317366 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:08.652753 21538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:08.653075 21538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:08.653291 21538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:08.660456 21538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:08.660561 21538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:08.660673 21538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:08.660775 21538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:08.660853 21538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:08.670730 21538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:08.672068 21538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:08.673404 21538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:08.685076 21543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:08.685111 21546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:08.685148 21544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:08.685988 21538 server_base.cc:1034] running on GCE node
I20260926 21:42:08.686830 21538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:08.688207 21538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:08.689402 21538 hybrid_clock.cc:648] HybridClock initialized: now 1790458928689368 us; error 30 us; skew 500 ppm
I20260926 21:42:08.689746 21538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:08 dist-test-slave-lhhq krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:08.699589 21538 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:42:08.701283 21538 webserver.cc:460] Webserver started at http://127.0.0.1:36523/ using document root <none> and password file <none>
I20260926 21:42:08.702013 21538 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:08.702145 21538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:08.702469 21538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:08.705631 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "16067f52795f473980d6c5fd35d5ce4e"
format_stamp: "Formatted at 2026-09-26 21:42:08 on dist-test-slave-lhhq"
server_key: "ab9449b4bd2d4078bb2c495fad5a8cd5"
server_key_iv: "7aed74bc5e838ed4ddb9eed60744c55e"
server_key_version: "encryptionkey@0"
I20260926 21:42:08.706421 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "16067f52795f473980d6c5fd35d5ce4e"
format_stamp: "Formatted at 2026-09-26 21:42:08 on dist-test-slave-lhhq"
server_key: "ab9449b4bd2d4078bb2c495fad5a8cd5"
server_key_iv: "7aed74bc5e838ed4ddb9eed60744c55e"
server_key_version: "encryptionkey@0"
I20260926 21:42:08.711572 21538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 21:42:08.715075 21553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:08.716473 21538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:42:08.716673 21538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "16067f52795f473980d6c5fd35d5ce4e"
format_stamp: "Formatted at 2026-09-26 21:42:08 on dist-test-slave-lhhq"
server_key: "ab9449b4bd2d4078bb2c495fad5a8cd5"
server_key_iv: "7aed74bc5e838ed4ddb9eed60744c55e"
server_key_version: "encryptionkey@0"
I20260926 21:42:08.716888 21538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:08.730195 21538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:08.734256 21538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:08.734563 21538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:08.754022 21538 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46499
I20260926 21:42:08.754046 21604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46499 every 8 connection(s)
I20260926 21:42:08.755527 21538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:08.759473 21605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:08.763567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 21538
I20260926 21:42:08.763783 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:08.764298 388 external_mini_cluster.cc:1428] Setting key 81be639e97076a52910663758770a6ff
I20260926 21:42:08.772670 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e: Bootstrap starting.
I20260926 21:42:08.776437 21605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:08.777752 21605 log.cc:824] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:08.778506 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:08.766167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:42183 (local address 127.0.0.1:46499)
0926 21:42:08.766988 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:08.767001 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:08.767042 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:42:08.768678 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:08.768709 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:08.769150 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:08.769572 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:08.770234 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:08.771861 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:08.773759 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:08.774442 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:08.774575 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:08.775894 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:08.775901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:08.775915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:08.776447 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:08.776557 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:08.776599 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:08.776762 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:08.777075 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 21:42:08.777280 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":351,"threads_started":1}
I20260926 21:42:08.780846 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e: No bootstrap required, opened a new log
I20260926 21:42:08.782548 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:08.789291 21605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16067f52795f473980d6c5fd35d5ce4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } }
I20260926 21:42:08.789783 21605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:08.789907 21605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16067f52795f473980d6c5fd35d5ce4e, State: Initialized, Role: FOLLOWER
I20260926 21:42:08.790576 21605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16067f52795f473980d6c5fd35d5ce4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } }
I20260926 21:42:08.790809 21605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:08.790951 21605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:08.791131 21605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:08.793972 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:08.786833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:36389 (local address 127.0.0.1:46499)
0926 21:42:08.787024 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:08.787032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:08.787502 (+ 470us) server_negotiation.cc:407] Connection header received
0926 21:42:08.787660 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:08.787671 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:08.787755 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:08.787864 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:08.788522 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:08.789642 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:08.791258 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:08.791944 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:08.792078 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:08.793156 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:08.793162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:08.793167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:08.793240 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:08.793301 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:08.793322 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:08.793432 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:08.793605 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 21:42:08.793659 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:42:08.794160 21605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16067f52795f473980d6c5fd35d5ce4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } }
I20260926 21:42:08.794720 21605 leader_election.cc:302] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16067f52795f473980d6c5fd35d5ce4e; no voters:
I20260926 21:42:08.795398 21605 leader_election.cc:288] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:08.795526 21615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:08.796247 21615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [term 1 LEADER]: Becoming Leader. State: Replica: 16067f52795f473980d6c5fd35d5ce4e, State: Running, Role: LEADER
I20260926 21:42:08.796790 21615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16067f52795f473980d6c5fd35d5ce4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } }
I20260926 21:42:08.797574 21605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:08.800204 21617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16067f52795f473980d6c5fd35d5ce4e. Latest consensus state: current_term: 1 leader_uuid: "16067f52795f473980d6c5fd35d5ce4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16067f52795f473980d6c5fd35d5ce4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } } }
I20260926 21:42:08.800110 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16067f52795f473980d6c5fd35d5ce4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16067f52795f473980d6c5fd35d5ce4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46499 } } }
I20260926 21:42:08.800596 21617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:08.800599 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:08.803175 21623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:08.808430 21623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:08.818176 21623 catalog_manager.cc:1347] Generated new cluster ID: 5dee45cce07f4e9da085d8dde4354452
I20260926 21:42:08.818313 21623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:08.835563 21623 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:08.836918 21623 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:08.846684 21623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16067f52795f473980d6c5fd35d5ce4e: Generated new TSK 0
I20260926 21:42:08.847618 21623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:08.858268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21538
W20260926 21:42:08.859401 21600 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.99:42183
W20260926 21:42:08.860139 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:08.860291 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 21538
W20260926 21:42:08.876988 21523 connection.cc:535] client connection to 127.0.0.1:46499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46499): Connection reset by peer (error 104)
2026-09-26T21:42:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:08 dist-test-slave-lhhq krb5kdc[21636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:08 dist-test-slave-lhhq krb5kdc[21636](info): set up 2 sockets
Sep 26 21:42:08 dist-test-slave-lhhq krb5kdc[21636](info): commencing operation
krb5kdc: starting...
W20260926 21:42:09.969805 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:10 dist-test-slave-lhhq krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:42:10.223960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:10.555548 21649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:10.555857 21649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:10.556046 21649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:10.563387 21649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:10.563515 21649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:10.563637 21649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:10.563738 21649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:10.563829 21649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:10.573843 21649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:10.575261 21649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:10.576615 21649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:10.588627 21655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:10.588572 21657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:10.588816 21654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:10.589506 21649 server_base.cc:1034] running on GCE node
I20260926 21:42:10.590341 21649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:10.591845 21649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:10.593075 21649 hybrid_clock.cc:648] HybridClock initialized: now 1790458930593024 us; error 49 us; skew 500 ppm
I20260926 21:42:10.593485 21649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:10 dist-test-slave-lhhq krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:10.596673 21649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:42:10.598240 21649 webserver.cc:460] Webserver started at http://127.0.0.1:44201/ using document root <none> and password file <none>
I20260926 21:42:10.598930 21649 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:10.599041 21649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:10.599331 21649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:10.602576 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "7968621dfe3a4f5897ce8d3d5ea47fda"
format_stamp: "Formatted at 2026-09-26 21:42:10 on dist-test-slave-lhhq"
server_key: "18e37ffd5f84dff202412069042ae16f"
server_key_iv: "72db698fdd46f15cabac80b85505519f"
server_key_version: "encryptionkey@0"
I20260926 21:42:10.603343 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "7968621dfe3a4f5897ce8d3d5ea47fda"
format_stamp: "Formatted at 2026-09-26 21:42:10 on dist-test-slave-lhhq"
server_key: "18e37ffd5f84dff202412069042ae16f"
server_key_iv: "72db698fdd46f15cabac80b85505519f"
server_key_version: "encryptionkey@0"
I20260926 21:42:10.608976 21649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 21:42:10.612547 21664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:10.614008 21649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:42:10.614203 21649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "7968621dfe3a4f5897ce8d3d5ea47fda"
format_stamp: "Formatted at 2026-09-26 21:42:10 on dist-test-slave-lhhq"
server_key: "18e37ffd5f84dff202412069042ae16f"
server_key_iv: "72db698fdd46f15cabac80b85505519f"
server_key_version: "encryptionkey@0"
I20260926 21:42:10.614415 21649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:10.620296 21649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:10.624567 21649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:10.624866 21649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:10.643874 21649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45053
I20260926 21:42:10.643915 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45053 every 8 connection(s)
I20260926 21:42:10.645327 21649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:10.648178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 21649
I20260926 21:42:10.648379 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:10.648878 388 external_mini_cluster.cc:1428] Setting key 32c955d775aef5d8286b0a432e00cb45
I20260926 21:42:10.649646 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:10.658237 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:10.650823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:42627 (local address 127.0.0.1:45053)
0926 21:42:10.651688 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:10.651701 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:10.651743 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:42:10.654437 (+ 2694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:10.654460 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:10.654849 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:10.655223 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:10.656534 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:10.656541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:10.656555 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:10.656574 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:10.656637 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:10.656665 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:10.656847 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:10.656995 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 21:42:10.657255 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":416,"threads_started":1}
I20260926 21:42:10.661909 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:10.664793 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda: Bootstrap starting.
I20260926 21:42:10.669505 21716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:10.670220 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:10.667579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:53409 (local address 127.0.0.1:45053)
0926 21:42:10.667773 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:10.667780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:10.668183 (+ 403us) server_negotiation.cc:407] Connection header received
0926 21:42:10.668330 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:10.668334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:10.668408 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:10.668534 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:10.669465 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:10.669472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:10.669479 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:10.669505 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:10.669552 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:10.669574 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:10.669694 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:10.669826 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 21:42:10.669899 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:42:10.670872 21716 log.cc:824] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:10.673775 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda: No bootstrap required, opened a new log
I20260926 21:42:10.680914 21716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7968621dfe3a4f5897ce8d3d5ea47fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } }
I20260926 21:42:10.681322 21716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:10.681454 21716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7968621dfe3a4f5897ce8d3d5ea47fda, State: Initialized, Role: FOLLOWER
I20260926 21:42:10.682010 21716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7968621dfe3a4f5897ce8d3d5ea47fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } }
I20260926 21:42:10.682247 21716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:10.682392 21716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:10.682566 21716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:10.684729 21716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7968621dfe3a4f5897ce8d3d5ea47fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } }
I20260926 21:42:10.685290 21716 leader_election.cc:302] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7968621dfe3a4f5897ce8d3d5ea47fda; no voters:
I20260926 21:42:10.685868 21716 leader_election.cc:288] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:10.686009 21726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:10.687826 21716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:10.688179 21726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [term 1 LEADER]: Becoming Leader. State: Replica: 7968621dfe3a4f5897ce8d3d5ea47fda, State: Running, Role: LEADER
I20260926 21:42:10.688675 21726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7968621dfe3a4f5897ce8d3d5ea47fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } }
I20260926 21:42:10.691593 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7968621dfe3a4f5897ce8d3d5ea47fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7968621dfe3a4f5897ce8d3d5ea47fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } } }
I20260926 21:42:10.691737 21728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7968621dfe3a4f5897ce8d3d5ea47fda. Latest consensus state: current_term: 1 leader_uuid: "7968621dfe3a4f5897ce8d3d5ea47fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7968621dfe3a4f5897ce8d3d5ea47fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } } }
I20260926 21:42:10.692005 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:10.692044 21728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:10.693475 21737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:10.699697 21737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:10.708635 21737 catalog_manager.cc:1347] Generated new cluster ID: e1b26a92330d49e382ec6bdb0fc00ab2
I20260926 21:42:10.708765 21737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:10.728060 21737 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:10.728597 21737 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:10.740039 21737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7968621dfe3a4f5897ce8d3d5ea47fda: Generated new TSK 0
I20260926 21:42:10.741232 21737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:10.752092 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21649
W20260926 21:42:10.753908 21711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.99:42627
W20260926 21:42:10.754587 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21649: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:10.754750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 21649
2026-09-26T21:42:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:10 dist-test-slave-lhhq krb5kdc[21747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:10 dist-test-slave-lhhq krb5kdc[21747](info): set up 2 sockets
Sep 26 21:42:10 dist-test-slave-lhhq krb5kdc[21747](info): commencing operation
krb5kdc: starting...
W20260926 21:42:11.861768 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:12 dist-test-slave-lhhq krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:42:12.120592 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:12.457967 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:12.458340 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:12.458524 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:12.466624 21760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:12.466842 21760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:12.466971 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:12.467093 21760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:12.467196 21760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:12.477950 21760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34597
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:12.479421 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:12.480782 21760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:12.493000 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:12.493057 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:12.493036 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:12.493957 21760 server_base.cc:1034] running on GCE node
I20260926 21:42:12.494823 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:12.496268 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:12.497493 21760 hybrid_clock.cc:648] HybridClock initialized: now 1790458932497461 us; error 59 us; skew 500 ppm
I20260926 21:42:12.497854 21760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:12 dist-test-slave-lhhq krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:12.500999 21760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:42:12.502497 21760 webserver.cc:460] Webserver started at http://127.0.0.1:43675/ using document root <none> and password file <none>
I20260926 21:42:12.503182 21760 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:12.503334 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:12.503636 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:12.506806 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "586b8bed95a44028afb27177016bccbd"
format_stamp: "Formatted at 2026-09-26 21:42:12 on dist-test-slave-lhhq"
server_key: "03a6effe832c4f192090077ae77bb4cb"
server_key_iv: "beaa4f525950a7a497c312d9d3c468db"
server_key_version: "encryptionkey@0"
I20260926 21:42:12.507552 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "586b8bed95a44028afb27177016bccbd"
format_stamp: "Formatted at 2026-09-26 21:42:12 on dist-test-slave-lhhq"
server_key: "03a6effe832c4f192090077ae77bb4cb"
server_key_iv: "beaa4f525950a7a497c312d9d3c468db"
server_key_version: "encryptionkey@0"
I20260926 21:42:12.512526 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 21:42:12.515954 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:12.517313 21760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 21:42:12.517493 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "586b8bed95a44028afb27177016bccbd"
format_stamp: "Formatted at 2026-09-26 21:42:12 on dist-test-slave-lhhq"
server_key: "03a6effe832c4f192090077ae77bb4cb"
server_key_iv: "beaa4f525950a7a497c312d9d3c468db"
server_key_version: "encryptionkey@0"
I20260926 21:42:12.517686 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:12.562182 21760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:12.566797 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:12.567096 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:12.585476 21760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34597
I20260926 21:42:12.585510 21826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34597 every 8 connection(s)
I20260926 21:42:12.586912 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:12.590617 21827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:12.596223 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 21760
I20260926 21:42:12.596432 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:12.596853 388 external_mini_cluster.cc:1428] Setting key 298cc5d4a90665330aba2d50cd519ee1
I20260926 21:42:12.601564 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd: Bootstrap starting.
I20260926 21:42:12.605314 21827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:12.606633 21827 log.cc:824] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:12.610281 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd: No bootstrap required, opened a new log
Sep 26 21:42:12 dist-test-slave-lhhq krb5kdc[21747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 21:42:12.619023 21827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b8bed95a44028afb27177016bccbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34597 } }
I20260926 21:42:12.619475 21827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:12.619617 21827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 586b8bed95a44028afb27177016bccbd, State: Initialized, Role: FOLLOWER
I20260926 21:42:12.620373 21827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b8bed95a44028afb27177016bccbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34597 } }
I20260926 21:42:12.620644 21827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:12.620846 21827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:12.621061 21827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:12.621541 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:12.598827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:47469 (local address 127.0.0.1:34597)
0926 21:42:12.599671 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:12.599684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:12.599725 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:42:12.601444 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:12.601467 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:12.601944 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:12.602278 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:12.604092 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:12.605456 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:12.607342 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:12.608339 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:12.608473 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:12.613617 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:12.613627 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:12.613647 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:12.613685 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:12.616911 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:12.617792 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:12.617799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:12.617806 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:12.618007 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:12.618473 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:12.618486 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:12.618489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:12.619011 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:12.619384 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:12.619856 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 21:42:12.620195 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":402,"threads_started":1}
I20260926 21:42:12.623737 21827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b8bed95a44028afb27177016bccbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34597 } }
I20260926 21:42:12.624306 21827 leader_election.cc:302] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 586b8bed95a44028afb27177016bccbd; no voters:
I20260926 21:42:12.625007 21827 leader_election.cc:288] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:12.625134 21832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:12.625694 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:12.627810 21827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:12.632310 21832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [term 1 LEADER]: Becoming Leader. State: Replica: 586b8bed95a44028afb27177016bccbd, State: Running, Role: LEADER
I20260926 21:42:12.633138 21832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b8bed95a44028afb27177016bccbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34597 } }
I20260926 21:42:12.638998 21833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "586b8bed95a44028afb27177016bccbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b8bed95a44028afb27177016bccbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34597 } } }
I20260926 21:42:12.639513 21833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:12.651996 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:12.630666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:43053 (local address 127.0.0.1:34597)
0926 21:42:12.630945 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:12.630950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:12.630971 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:42:12.631173 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:12.631178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:12.631260 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:12.631318 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:12.632925 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:12.634328 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:12.636632 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:12.637459 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:12.637587 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:12.641309 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:12.641318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:12.641326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:12.641383 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:12.646302 (+ 4919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:12.647659 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:12.647668 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:12.647674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:12.647866 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:12.648646 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:12.648654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:12.648660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:12.649206 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:12.650754 (+ 1548us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:12.650984 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 21:42:12.651318 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260926 21:42:12.654049 21834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 586b8bed95a44028afb27177016bccbd. Latest consensus state: current_term: 1 leader_uuid: "586b8bed95a44028afb27177016bccbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b8bed95a44028afb27177016bccbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34597 } } }
I20260926 21:42:12.654464 21834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:12.654872 21847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:12.659783 21847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:12.668910 21847 catalog_manager.cc:1347] Generated new cluster ID: 8879797892114ca7bfc66522d92e2660
I20260926 21:42:12.669023 21847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:12.699090 21847 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:12.700448 21847 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:12.711625 21847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 586b8bed95a44028afb27177016bccbd: Generated new TSK 0
I20260926 21:42:12.712483 21847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:12.728874 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21760
I20260926 21:42:12.730053 21822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:12.824913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 21760
W20260926 21:42:12.842332 21745 connection.cc:535] client connection to 127.0.0.1:34597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34597): Connection reset by peer (error 104)
2026-09-26T21:42:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:12 dist-test-slave-lhhq krb5kdc[21859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:12 dist-test-slave-lhhq krb5kdc[21859](info): set up 2 sockets
Sep 26 21:42:12 dist-test-slave-lhhq krb5kdc[21859](info): commencing operation
krb5kdc: starting...
W20260926 21:42:13.933596 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:14 dist-test-slave-lhhq krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:42:14.192862 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:14.521524 21872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:14.521797 21872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:14.522001 21872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:14.529018 21872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:14.529147 21872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:14.529270 21872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:14.529368 21872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:14.529433 21872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:14.539314 21872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:14.540664 21872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:14.542011 21872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:14.553505 21878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:14.553566 21880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:14.553612 21877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:14.554387 21872 server_base.cc:1034] running on GCE node
I20260926 21:42:14.555163 21872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:14.556485 21872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:14.557703 21872 hybrid_clock.cc:648] HybridClock initialized: now 1790458934557664 us; error 51 us; skew 500 ppm
I20260926 21:42:14.558043 21872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:14 dist-test-slave-lhhq krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:14.567703 21872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:42:14.569361 21872 webserver.cc:460] Webserver started at http://127.0.0.1:45151/ using document root <none> and password file <none>
I20260926 21:42:14.570076 21872 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:14.570222 21872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:14.570537 21872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:14.573611 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "e56c8da248254a0691fa5f7c5e5bf24e"
format_stamp: "Formatted at 2026-09-26 21:42:14 on dist-test-slave-lhhq"
server_key: "021b551e68b4a1c91217e4f4d6a94c7b"
server_key_iv: "ad1b19fa115337b3a83b5f9ece167d21"
server_key_version: "encryptionkey@0"
I20260926 21:42:14.574370 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "e56c8da248254a0691fa5f7c5e5bf24e"
format_stamp: "Formatted at 2026-09-26 21:42:14 on dist-test-slave-lhhq"
server_key: "021b551e68b4a1c91217e4f4d6a94c7b"
server_key_iv: "ad1b19fa115337b3a83b5f9ece167d21"
server_key_version: "encryptionkey@0"
I20260926 21:42:14.579442 21872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 21:42:14.582834 21887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:14.584162 21872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 21:42:14.584348 21872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "e56c8da248254a0691fa5f7c5e5bf24e"
format_stamp: "Formatted at 2026-09-26 21:42:14 on dist-test-slave-lhhq"
server_key: "021b551e68b4a1c91217e4f4d6a94c7b"
server_key_iv: "ad1b19fa115337b3a83b5f9ece167d21"
server_key_version: "encryptionkey@0"
I20260926 21:42:14.584544 21872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:14.600234 21872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:14.604671 21872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:14.604962 21872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:14.623572 21872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41055
I20260926 21:42:14.623680 21938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41055 every 8 connection(s)
I20260926 21:42:14.624960 21872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:14.627570 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 21872
I20260926 21:42:14.627763 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:14.628284 388 external_mini_cluster.cc:1428] Setting key 28317f34429e8be3383dcedefc836651
I20260926 21:42:14.629248 21939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:14.641067 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e: Bootstrap starting.
I20260926 21:42:14.642302 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:14.630191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:54119 (local address 127.0.0.1:41055)
0926 21:42:14.630936 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:14.630948 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:14.630987 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:42:14.632508 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:14.632528 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:14.632918 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:14.633361 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:14.633902 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:14.635536 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:14.637114 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:14.637939 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:14.638210 (+ 271us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:14.639750 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:14.639759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:14.639774 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:14.640366 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:14.640448 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:14.640475 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:14.640705 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:14.641010 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 21:42:14.641252 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":336,"threads_started":1}
I20260926 21:42:14.644163 21939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:14.645392 21939 log.cc:824] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:14.646535 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:14.648022 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e: No bootstrap required, opened a new log
I20260926 21:42:14.656406 21939 raft_consensus.cc:348] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56c8da248254a0691fa5f7c5e5bf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260926 21:42:14.656857 21939 raft_consensus.cc:374] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:14.656980 21939 raft_consensus.cc:729] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56c8da248254a0691fa5f7c5e5bf24e, State: Initialized, Role: FOLLOWER
I20260926 21:42:14.657713 21939 consensus_queue.cc:260] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56c8da248254a0691fa5f7c5e5bf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260926 21:42:14.657999 21939 raft_consensus.cc:388] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:14.658143 21939 raft_consensus.cc:482] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:14.658344 21939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:14.659197 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:14.651569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:34473 (local address 127.0.0.1:41055)
0926 21:42:14.651765 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:14.651772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:14.652260 (+ 488us) server_negotiation.cc:407] Connection header received
0926 21:42:14.652424 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:14.652429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:14.652504 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:14.652605 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:14.653227 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:14.654412 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:14.655978 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:14.656601 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:14.656737 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:14.658347 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:14.658353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:14.658364 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:14.658382 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:14.658440 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:14.658463 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:14.658568 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:14.658798 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 21:42:14.658837 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:42:14.661345 21939 raft_consensus.cc:504] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56c8da248254a0691fa5f7c5e5bf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260926 21:42:14.661893 21939 leader_election.cc:302] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e56c8da248254a0691fa5f7c5e5bf24e; no voters:
I20260926 21:42:14.662472 21939 leader_election.cc:288] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:14.662619 21949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:14.663337 21949 raft_consensus.cc:686] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [term 1 LEADER]: Becoming Leader. State: Replica: e56c8da248254a0691fa5f7c5e5bf24e, State: Running, Role: LEADER
I20260926 21:42:14.663923 21949 consensus_queue.cc:237] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56c8da248254a0691fa5f7c5e5bf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260926 21:42:14.664539 21939 sys_catalog.cc:564] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:14.666761 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e56c8da248254a0691fa5f7c5e5bf24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56c8da248254a0691fa5f7c5e5bf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } } }
I20260926 21:42:14.666862 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e56c8da248254a0691fa5f7c5e5bf24e. Latest consensus state: current_term: 1 leader_uuid: "e56c8da248254a0691fa5f7c5e5bf24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56c8da248254a0691fa5f7c5e5bf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } } }
I20260926 21:42:14.667208 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:14.667208 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:14.669483 21956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:14.674001 21956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:14.682166 21956 catalog_manager.cc:1347] Generated new cluster ID: 5c8f8ed00c4448a5a52ad4664ea7d5c9
I20260926 21:42:14.682315 21956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:14.718860 21956 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:14.720757 21956 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:14.728242 21956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e56c8da248254a0691fa5f7c5e5bf24e: Generated new TSK 0
I20260926 21:42:14.729274 21956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:14.741570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21872
W20260926 21:42:14.742725 21934 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.99:54119
W20260926 21:42:14.743491 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21872: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:14.743654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 21872
W20260926 21:42:14.760571 21857 connection.cc:535] client connection to 127.0.0.1:41055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41055): Connection reset by peer (error 104)
2026-09-26T21:42:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1918 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:14 dist-test-slave-lhhq krb5kdc[21970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:14 dist-test-slave-lhhq krb5kdc[21970](info): set up 2 sockets
Sep 26 21:42:14 dist-test-slave-lhhq krb5kdc[21970](info): commencing operation
krb5kdc: starting...
W20260926 21:42:15.853585 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:16 dist-test-slave-lhhq krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 21:42:16.114668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:16.448726 21983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:16.449025 21983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:16.449118 21983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:16.456413 21983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:16.456514 21983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:16.456559 21983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:16.456619 21983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:16.456663 21983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:16.467109 21983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:16.468425 21983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:16.469871 21983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:16.481978 21989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:16.482054 21988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:16.482035 21991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:16.483218 21983 server_base.cc:1034] running on GCE node
I20260926 21:42:16.484066 21983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:16.485538 21983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:16.486752 21983 hybrid_clock.cc:648] HybridClock initialized: now 1790458936486704 us; error 35 us; skew 500 ppm
I20260926 21:42:16.487139 21983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:16 dist-test-slave-lhhq krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:16.490381 21983 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 21:42:16.491775 21983 webserver.cc:460] Webserver started at http://127.0.0.1:43321/ using document root <none> and password file <none>
I20260926 21:42:16.492463 21983 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:16.492597 21983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:16.492913 21983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:16.495898 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "c800dac2b386488ea291c972ab3fbb71"
format_stamp: "Formatted at 2026-09-26 21:42:16 on dist-test-slave-lhhq"
server_key: "580c336b4955414363a9c762a2e2518f"
server_key_iv: "864d97e7f036c7228af0a48298b43a92"
server_key_version: "encryptionkey@0"
I20260926 21:42:16.496650 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "c800dac2b386488ea291c972ab3fbb71"
format_stamp: "Formatted at 2026-09-26 21:42:16 on dist-test-slave-lhhq"
server_key: "580c336b4955414363a9c762a2e2518f"
server_key_iv: "864d97e7f036c7228af0a48298b43a92"
server_key_version: "encryptionkey@0"
I20260926 21:42:16.501817 21983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 21:42:16.505110 21998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:16.506428 21983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:42:16.506618 21983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "c800dac2b386488ea291c972ab3fbb71"
format_stamp: "Formatted at 2026-09-26 21:42:16 on dist-test-slave-lhhq"
server_key: "580c336b4955414363a9c762a2e2518f"
server_key_iv: "864d97e7f036c7228af0a48298b43a92"
server_key_version: "encryptionkey@0"
I20260926 21:42:16.506825 21983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:16.512576 21983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:16.516886 21983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:16.517210 21983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:16.535754 21983 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36153
I20260926 21:42:16.535804 22049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36153 every 8 connection(s)
I20260926 21:42:16.537166 21983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:16.540216 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 21983
I20260926 21:42:16.540408 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:16.540964 388 external_mini_cluster.cc:1428] Setting key 72261941637f6b694983ed4888c87ba5
I20260926 21:42:16.541329 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:16.550166 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:16.542849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:51363 (local address 127.0.0.1:36153)
0926 21:42:16.543655 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:16.543673 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:16.543726 (+ 53us) server_negotiation.cc:407] Connection header received
0926 21:42:16.546141 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:16.546172 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:16.546648 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:16.547121 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:16.548130 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:16.548139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:16.548155 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:16.548172 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:16.548256 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:16.548283 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:16.548518 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:16.548646 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 21:42:16.548881 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":323,"threads_started":1}
I20260926 21:42:16.553874 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:16.556259 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71: Bootstrap starting.
I20260926 21:42:16.560454 22050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:16.561983 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:16.558996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:41875 (local address 127.0.0.1:36153)
0926 21:42:16.559211 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:16.559221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:16.559760 (+ 539us) server_negotiation.cc:407] Connection header received
0926 21:42:16.559900 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:16.559904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:16.559956 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:16.560054 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:16.561134 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:16.561138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:16.561141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:16.561157 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:16.561268 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:16.561296 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:16.561404 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:16.561577 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 21:42:16.561629 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 21:42:16.561962 22050 log.cc:824] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:16.564910 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71: No bootstrap required, opened a new log
I20260926 21:42:16.572176 22050 raft_consensus.cc:348] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c800dac2b386488ea291c972ab3fbb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } }
I20260926 21:42:16.572583 22050 raft_consensus.cc:374] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:16.572688 22050 raft_consensus.cc:729] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c800dac2b386488ea291c972ab3fbb71, State: Initialized, Role: FOLLOWER
I20260926 21:42:16.573423 22050 consensus_queue.cc:260] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c800dac2b386488ea291c972ab3fbb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } }
I20260926 21:42:16.573704 22050 raft_consensus.cc:388] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:16.573868 22050 raft_consensus.cc:482] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:16.574034 22050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:16.576195 22050 raft_consensus.cc:504] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c800dac2b386488ea291c972ab3fbb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } }
I20260926 21:42:16.576735 22050 leader_election.cc:302] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c800dac2b386488ea291c972ab3fbb71; no voters:
I20260926 21:42:16.577417 22050 leader_election.cc:288] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:16.577574 22060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:16.578608 22060 raft_consensus.cc:686] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [term 1 LEADER]: Becoming Leader. State: Replica: c800dac2b386488ea291c972ab3fbb71, State: Running, Role: LEADER
I20260926 21:42:16.579180 22060 consensus_queue.cc:237] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c800dac2b386488ea291c972ab3fbb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } }
I20260926 21:42:16.579720 22050 sys_catalog.cc:564] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:16.582293 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c800dac2b386488ea291c972ab3fbb71. Latest consensus state: current_term: 1 leader_uuid: "c800dac2b386488ea291c972ab3fbb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c800dac2b386488ea291c972ab3fbb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } } }
I20260926 21:42:16.582379 22061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c800dac2b386488ea291c972ab3fbb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c800dac2b386488ea291c972ab3fbb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } } }
I20260926 21:42:16.582744 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:16.582809 22061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:16.585997 22067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:16.591586 22067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:16.601912 22067 catalog_manager.cc:1347] Generated new cluster ID: 08bfae7c681248f18fc3c694b3f6fffd
I20260926 21:42:16.602062 22067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:16.621793 22067 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:16.622233 22067 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:16.635393 22067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c800dac2b386488ea291c972ab3fbb71: Generated new TSK 0
I20260926 21:42:16.636245 22067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:16.642884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21983
W20260926 21:42:16.644481 22045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.99:51363
W20260926 21:42:16.645273 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 21983: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:16.645437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 21983
2026-09-26T21:42:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1906 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:16 dist-test-slave-lhhq krb5kdc[22081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:16 dist-test-slave-lhhq krb5kdc[22081](info): set up 2 sockets
Sep 26 21:42:16 dist-test-slave-lhhq krb5kdc[22081](info): commencing operation
krb5kdc: starting...
W20260926 21:42:17.765558 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:17 dist-test-slave-lhhq krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:42:18.020432 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36437
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:18.353987 22094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:18.354300 22094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:18.354494 22094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:18.361913 22094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:18.362043 22094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:18.362190 22094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:18.362287 22094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:18.362375 22094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:18.373973 22094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:18.375450 22094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:18.376827 22094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:18.389387 22099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:18.389321 22100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:18.389351 22102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:18.390185 22094 server_base.cc:1034] running on GCE node
I20260926 21:42:18.390998 22094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:18.392745 22094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:18.394016 22094 hybrid_clock.cc:648] HybridClock initialized: now 1790458938393954 us; error 54 us; skew 500 ppm
I20260926 21:42:18.394394 22094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:18 dist-test-slave-lhhq krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:18.397617 22094 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:42:18.399274 22094 webserver.cc:460] Webserver started at http://127.0.97.62:40911/ using document root <none> and password file <none>
I20260926 21:42:18.399962 22094 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:18.400065 22094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:18.400352 22094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:18.403645 22094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "771c97fab4a642beb3850ae6c13f8e5a"
format_stamp: "Formatted at 2026-09-26 21:42:18 on dist-test-slave-lhhq"
server_key: "1ef954fae77e2b5cf18f4ba1e9561deb"
server_key_iv: "5e8f37354eecaa655694705aa0f0b396"
server_key_version: "encryptionkey@0"
I20260926 21:42:18.404402 22094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "771c97fab4a642beb3850ae6c13f8e5a"
format_stamp: "Formatted at 2026-09-26 21:42:18 on dist-test-slave-lhhq"
server_key: "1ef954fae77e2b5cf18f4ba1e9561deb"
server_key_iv: "5e8f37354eecaa655694705aa0f0b396"
server_key_version: "encryptionkey@0"
I20260926 21:42:18.409793 22094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 21:42:18.413352 22109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:18.414693 22094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:42:18.414907 22094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "771c97fab4a642beb3850ae6c13f8e5a"
format_stamp: "Formatted at 2026-09-26 21:42:18 on dist-test-slave-lhhq"
server_key: "1ef954fae77e2b5cf18f4ba1e9561deb"
server_key_iv: "5e8f37354eecaa655694705aa0f0b396"
server_key_version: "encryptionkey@0"
I20260926 21:42:18.415110 22094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:18.446331 22094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:18.450806 22094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:18.451118 22094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:18.470329 22094 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36437
I20260926 21:42:18.470359 22160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36437 every 8 connection(s)
I20260926 21:42:18.471895 22094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:18.475739 22161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:18.475776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 22094
I20260926 21:42:18.475965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:18.476429 388 external_mini_cluster.cc:1428] Setting key 34d37ed0cd540176dba5618bc37c37c1
I20260926 21:42:18.490311 22161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a: Bootstrap starting.
Sep 26 21:42:18 dist-test-slave-lhhq krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:42:18.493664 22161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:18.494769 22161 log.cc:824] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:18.498373 22161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a: No bootstrap required, opened a new log
I20260926 21:42:18.500787 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:18.478190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41829 (local address 127.0.97.62:36437)
0926 21:42:18.479033 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:18.479045 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:18.479088 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:42:18.480639 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:18.480667 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:18.481073 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:18.481491 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:18.483335 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:18.484756 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:18.486642 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:18.487342 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:18.487483 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:18.492307 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:18.492315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:18.492330 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:18.492369 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:18.495729 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:18.496755 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:18.496764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:18.496773 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:18.496990 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:18.497541 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:18.497546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:18.497549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:18.498206 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:18.498532 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:18.499041 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 21:42:18.499394 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":399,"threads_started":1}
I20260926 21:42:18.504967 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:18.508240 22161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c97fab4a642beb3850ae6c13f8e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36437 } }
I20260926 21:42:18.508790 22161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:18.508919 22161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771c97fab4a642beb3850ae6c13f8e5a, State: Initialized, Role: FOLLOWER
I20260926 21:42:18.509671 22161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c97fab4a642beb3850ae6c13f8e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36437 } }
I20260926 21:42:18.509984 22161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:18.510135 22161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:18.510349 22161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:18.513033 22161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c97fab4a642beb3850ae6c13f8e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36437 } }
I20260926 21:42:18.513655 22161 leader_election.cc:302] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 771c97fab4a642beb3850ae6c13f8e5a; no voters:
I20260926 21:42:18.514391 22161 leader_election.cc:288] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:18.514495 22171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:18.517132 22161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:18.521217 22171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [term 1 LEADER]: Becoming Leader. State: Replica: 771c97fab4a642beb3850ae6c13f8e5a, State: Running, Role: LEADER
I20260926 21:42:18.522004 22171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c97fab4a642beb3850ae6c13f8e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36437 } }
I20260926 21:42:18.525815 22172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "771c97fab4a642beb3850ae6c13f8e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c97fab4a642beb3850ae6c13f8e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36437 } } }
I20260926 21:42:18.526292 22172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:18.526717 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:18.510326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57021 (local address 127.0.97.62:36437)
0926 21:42:18.510532 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:18.510538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:18.511013 (+ 475us) server_negotiation.cc:407] Connection header received
0926 21:42:18.511121 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:18.511127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:18.511215 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:18.511333 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:18.512613 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:18.513862 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:18.515534 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:18.516084 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:18.516163 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:18.519297 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:18.519304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:18.519308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:18.519355 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:18.522512 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:18.523343 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:18.523349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:18.523353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:18.523493 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:18.523941 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:18.523949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:18.523952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:18.524408 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:18.524701 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:18.526010 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0926 21:42:18.526192 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 21:42:18.534585 22180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:18.534615 22173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 771c97fab4a642beb3850ae6c13f8e5a. Latest consensus state: current_term: 1 leader_uuid: "771c97fab4a642beb3850ae6c13f8e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c97fab4a642beb3850ae6c13f8e5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36437 } } }
I20260926 21:42:18.534945 22173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:18.539959 22180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:18.549302 22180 catalog_manager.cc:1347] Generated new cluster ID: df1c3682356941729fe1ecf71a10efcf
I20260926 21:42:18.549420 22180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:18.564213 22180 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:18.565631 22180 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:18.573282 22180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 771c97fab4a642beb3850ae6c13f8e5a: Generated new TSK 0
I20260926 21:42:18.574152 22180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:18.582000 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22094
I20260926 21:42:18.583339 22156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:18.674582 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 22094
W20260926 21:42:18.691903 22079 connection.cc:535] client connection to 127.0.97.62:36437 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:36437): Connection reset by peer (error 104)
2026-09-26T21:42:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2025 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:18 dist-test-slave-lhhq krb5kdc[22194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:18 dist-test-slave-lhhq krb5kdc[22194](info): set up 2 sockets
Sep 26 21:42:18 dist-test-slave-lhhq krb5kdc[22194](info): commencing operation
krb5kdc: starting...
W20260926 21:42:19.793712 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:19 dist-test-slave-lhhq krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:42:20.048532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:20.382107 22207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:20.382561 22207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:20.382786 22207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:20.390206 22207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:20.390324 22207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:20.390372 22207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:20.390432 22207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:20.390476 22207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:20.400418 22207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42219
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:20.401800 22207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:20.403090 22207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:20.414937 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:20.415016 22212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:20.414994 22213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:20.416324 22207 server_base.cc:1034] running on GCE node
I20260926 21:42:20.417098 22207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:20.418653 22207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:20.419909 22207 hybrid_clock.cc:648] HybridClock initialized: now 1790458940419854 us; error 52 us; skew 500 ppm
I20260926 21:42:20.420306 22207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:20 dist-test-slave-lhhq krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:20.434451 22207 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:42:20.436170 22207 webserver.cc:460] Webserver started at http://127.0.97.62:37709/ using document root <none> and password file <none>
I20260926 21:42:20.436956 22207 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:20.437124 22207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:20.437489 22207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:20.440618 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "a62a639ce89f43fcbce3449b800a2fc9"
format_stamp: "Formatted at 2026-09-26 21:42:20 on dist-test-slave-lhhq"
server_key: "dfb9a65b0b9f3a1938e62bf81fef4860"
server_key_iv: "ef2bfd111c8c59923949510f8f83e861"
server_key_version: "encryptionkey@0"
I20260926 21:42:20.441510 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "a62a639ce89f43fcbce3449b800a2fc9"
format_stamp: "Formatted at 2026-09-26 21:42:20 on dist-test-slave-lhhq"
server_key: "dfb9a65b0b9f3a1938e62bf81fef4860"
server_key_iv: "ef2bfd111c8c59923949510f8f83e861"
server_key_version: "encryptionkey@0"
I20260926 21:42:20.446908 22207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 21:42:20.450660 22222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:20.452028 22207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:42:20.452215 22207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "a62a639ce89f43fcbce3449b800a2fc9"
format_stamp: "Formatted at 2026-09-26 21:42:20 on dist-test-slave-lhhq"
server_key: "dfb9a65b0b9f3a1938e62bf81fef4860"
server_key_iv: "ef2bfd111c8c59923949510f8f83e861"
server_key_version: "encryptionkey@0"
I20260926 21:42:20.452420 22207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:20.477876 22207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:20.482540 22207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:20.482851 22207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:20.501874 22207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42219
I20260926 21:42:20.501904 22273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42219 every 8 connection(s)
I20260926 21:42:20.503345 22207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:20.507097 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:20.512945 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 22207
I20260926 21:42:20.513149 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:20.513597 388 external_mini_cluster.cc:1428] Setting key f5938c7121b5103312cc01d235c5624a
I20260926 21:42:20.518478 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9: Bootstrap starting.
I20260926 21:42:20.521715 22274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:20.522733 22274 log.cc:824] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:20.525830 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9: No bootstrap required, opened a new log
I20260926 21:42:20.527215 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:20.515557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33813 (local address 127.0.97.62:42219)
0926 21:42:20.516440 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:20.516456 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:20.516499 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:42:20.518272 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:20.518297 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:20.518761 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:20.519159 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:20.519701 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:20.521084 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:20.522543 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:20.523428 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:20.523630 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:20.524870 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:20.524878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:20.524897 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:20.525510 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:20.525585 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:20.525614 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:20.525786 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:20.526075 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 21:42:20.526242 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":429,"threads_started":1}
I20260926 21:42:20.531286 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:20.534919 22274 raft_consensus.cc:348] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62a639ce89f43fcbce3449b800a2fc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42219 } }
I20260926 21:42:20.535331 22274 raft_consensus.cc:374] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:20.535459 22274 raft_consensus.cc:729] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a639ce89f43fcbce3449b800a2fc9, State: Initialized, Role: FOLLOWER
I20260926 21:42:20.536188 22274 consensus_queue.cc:260] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62a639ce89f43fcbce3449b800a2fc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42219 } }
I20260926 21:42:20.536500 22274 raft_consensus.cc:388] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:20.536706 22274 raft_consensus.cc:482] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:20.536933 22274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:20.540005 22274 raft_consensus.cc:504] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62a639ce89f43fcbce3449b800a2fc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42219 } }
I20260926 21:42:20.540635 22274 leader_election.cc:302] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a62a639ce89f43fcbce3449b800a2fc9; no voters:
I20260926 21:42:20.541375 22274 leader_election.cc:288] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:20.541460 22284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:20.543490 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:20.535485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34125 (local address 127.0.97.62:42219)
0926 21:42:20.535668 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:20.535676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:20.536350 (+ 674us) server_negotiation.cc:407] Connection header received
0926 21:42:20.536782 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:20.536787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:20.536876 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:20.536973 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:20.537641 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:20.538424 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:20.540340 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:20.540833 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:20.540940 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:20.542436 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:20.542445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:20.542453 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:20.542488 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:20.542554 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:20.542588 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:20.542825 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:20.543094 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 21:42:20.543152 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:42:20.544229 22274 sys_catalog.cc:564] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:20.548455 22284 raft_consensus.cc:686] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [term 1 LEADER]: Becoming Leader. State: Replica: a62a639ce89f43fcbce3449b800a2fc9, State: Running, Role: LEADER
I20260926 21:42:20.549211 22284 consensus_queue.cc:237] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62a639ce89f43fcbce3449b800a2fc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42219 } }
I20260926 21:42:20.552347 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a62a639ce89f43fcbce3449b800a2fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62a639ce89f43fcbce3449b800a2fc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42219 } } }
I20260926 21:42:20.552825 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:20.552448 22286 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a62a639ce89f43fcbce3449b800a2fc9. Latest consensus state: current_term: 1 leader_uuid: "a62a639ce89f43fcbce3449b800a2fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62a639ce89f43fcbce3449b800a2fc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42219 } } }
I20260926 21:42:20.553939 22286 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:20.554137 22299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:20.558616 22299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:20.567080 22299 catalog_manager.cc:1347] Generated new cluster ID: 763aa2f5acb84dd2abba65bfd48aa3ab
I20260926 21:42:20.567202 22299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:20.574873 22299 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:20.576382 22299 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:20.584846 22299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a62a639ce89f43fcbce3449b800a2fc9: Generated new TSK 0
I20260926 21:42:20.585711 22299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:20.592701 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22207
W20260926 21:42:20.594184 22269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33813
W20260926 21:42:20.595019 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22207: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:20.595170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 22207
W20260926 21:42:20.611877 22192 connection.cc:535] client connection to 127.0.97.62:42219 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:42219): Connection reset by peer (error 104)
2026-09-26T21:42:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:20 dist-test-slave-lhhq krb5kdc[22305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:20 dist-test-slave-lhhq krb5kdc[22305](info): set up 2 sockets
Sep 26 21:42:20 dist-test-slave-lhhq krb5kdc[22305](info): commencing operation
krb5kdc: starting...
W20260926 21:42:21.701766 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:21 dist-test-slave-lhhq krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:42:21.964468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33715
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:22.292805 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:22.293231 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:22.293409 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:22.300789 22318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:22.300940 22318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:22.301045 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:22.301167 22318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:22.301273 22318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:22.311124 22318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33715
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:22.312426 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:22.313838 22318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:22.325816 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:22.325757 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:22.326458 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:22.326897 22318 server_base.cc:1034] running on GCE node
I20260926 21:42:22.327780 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:22.329270 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:22.330502 22318 hybrid_clock.cc:648] HybridClock initialized: now 1790458942330449 us; error 47 us; skew 500 ppm
I20260926 21:42:22.330885 22318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:22 dist-test-slave-lhhq krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:22.334076 22318 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:42:22.335608 22318 webserver.cc:460] Webserver started at http://127.0.97.62:39795/ using document root <none> and password file <none>
I20260926 21:42:22.336309 22318 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:22.336455 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:22.336771 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:22.339939 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "0bcf7130cd534527a569f6a54b04818a"
format_stamp: "Formatted at 2026-09-26 21:42:22 on dist-test-slave-lhhq"
server_key: "c4909ad5c636e38537e18658caac67ea"
server_key_iv: "df91b966eb262cd232ab5a31ab8ae528"
server_key_version: "encryptionkey@0"
I20260926 21:42:22.340705 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "0bcf7130cd534527a569f6a54b04818a"
format_stamp: "Formatted at 2026-09-26 21:42:22 on dist-test-slave-lhhq"
server_key: "c4909ad5c636e38537e18658caac67ea"
server_key_iv: "df91b966eb262cd232ab5a31ab8ae528"
server_key_version: "encryptionkey@0"
I20260926 21:42:22.346302 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:42:22.349985 22333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:22.351402 22318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:42:22.351600 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "0bcf7130cd534527a569f6a54b04818a"
format_stamp: "Formatted at 2026-09-26 21:42:22 on dist-test-slave-lhhq"
server_key: "c4909ad5c636e38537e18658caac67ea"
server_key_iv: "df91b966eb262cd232ab5a31ab8ae528"
server_key_version: "encryptionkey@0"
I20260926 21:42:22.351816 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:22.357398 22318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:22.361750 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:22.362044 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:22.380553 22318 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33715
I20260926 21:42:22.380599 22384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33715 every 8 connection(s)
I20260926 21:42:22.382126 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:22.386211 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:22.388880 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 22318
I20260926 21:42:22.389065 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:22.389700 388 external_mini_cluster.cc:1428] Setting key eebab0ffec1cc9af1dcbac72e0864dc0
I20260926 21:42:22.398623 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:22.392551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34557 (local address 127.0.97.62:33715)
0926 21:42:22.393405 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:22.393417 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:22.393459 (+ 42us) server_negotiation.cc:407] Connection header received
0926 21:42:22.395126 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:22.395149 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:22.395490 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:22.395892 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:22.396961 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:22.396967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:22.396983 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:22.396996 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:22.397053 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:22.397083 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:22.397343 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:22.397496 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 21:42:22.397675 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":343,"threads_started":1}
I20260926 21:42:22.400296 22385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a: Bootstrap starting.
I20260926 21:42:22.402923 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:22.404450 22385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:22.405965 22385 log.cc:824] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:22.409416 22385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a: No bootstrap required, opened a new log
I20260926 21:42:22.411352 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:22.408519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57067 (local address 127.0.97.62:33715)
0926 21:42:22.408723 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:22.408727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:22.409249 (+ 522us) server_negotiation.cc:407] Connection header received
0926 21:42:22.409483 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:22.409489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:22.409548 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:22.409660 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:22.410652 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:22.410659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:22.410663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:22.410689 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:22.410737 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:22.410760 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:22.410878 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:22.410991 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 21:42:22.411054 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 21:42:22.417662 22385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcf7130cd534527a569f6a54b04818a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33715 } }
I20260926 21:42:22.418040 22385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:22.418155 22385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcf7130cd534527a569f6a54b04818a, State: Initialized, Role: FOLLOWER
I20260926 21:42:22.418815 22385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcf7130cd534527a569f6a54b04818a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33715 } }
I20260926 21:42:22.419096 22385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:22.419277 22385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:22.419468 22385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:22.421723 22385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcf7130cd534527a569f6a54b04818a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33715 } }
I20260926 21:42:22.422225 22385 leader_election.cc:302] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bcf7130cd534527a569f6a54b04818a; no voters:
I20260926 21:42:22.422832 22385 leader_election.cc:288] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:22.422999 22395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:22.423676 22395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [term 1 LEADER]: Becoming Leader. State: Replica: 0bcf7130cd534527a569f6a54b04818a, State: Running, Role: LEADER
I20260926 21:42:22.424235 22395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcf7130cd534527a569f6a54b04818a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33715 } }
I20260926 21:42:22.425016 22385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:22.427335 22397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bcf7130cd534527a569f6a54b04818a. Latest consensus state: current_term: 1 leader_uuid: "0bcf7130cd534527a569f6a54b04818a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcf7130cd534527a569f6a54b04818a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33715 } } }
I20260926 21:42:22.427861 22397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:22.427366 22396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bcf7130cd534527a569f6a54b04818a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcf7130cd534527a569f6a54b04818a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33715 } } }
I20260926 21:42:22.428391 22396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:22.429761 22402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:22.434842 22402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:22.444290 22402 catalog_manager.cc:1347] Generated new cluster ID: 405044b5eabb402f9d07e6bf4c0ea500
I20260926 21:42:22.444437 22402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:22.462006 22402 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:22.462541 22402 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:22.479099 22402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bcf7130cd534527a569f6a54b04818a: Generated new TSK 0
I20260926 21:42:22.480285 22402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:22.490028 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22318
W20260926 21:42:22.491544 22380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34557
W20260926 21:42:22.492322 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22318: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:22.492482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 22318
2026-09-26T21:42:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:22 dist-test-slave-lhhq krb5kdc[22416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:22 dist-test-slave-lhhq krb5kdc[22416](info): set up 2 sockets
Sep 26 21:42:22 dist-test-slave-lhhq krb5kdc[22416](info): commencing operation
krb5kdc: starting...
W20260926 21:42:23.601779 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:23 dist-test-slave-lhhq krb5kdc[22416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:42:23.870131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38683
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:24.217690 22429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:24.218055 22429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:24.218225 22429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:24.225520 22429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:24.225698 22429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:24.225808 22429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:24.225914 22429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:24.225994 22429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:24.236073 22429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:24.237490 22429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:24.238864 22429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:24.250725 22437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:24.250682 22434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:24.250800 22435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:24.251570 22429 server_base.cc:1034] running on GCE node
I20260926 21:42:24.252277 22429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:24.253737 22429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:24.254931 22429 hybrid_clock.cc:648] HybridClock initialized: now 1790458944254876 us; error 48 us; skew 500 ppm
I20260926 21:42:24.255230 22429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:24 dist-test-slave-lhhq krb5kdc[22416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:24.258376 22429 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:42:24.259881 22429 webserver.cc:460] Webserver started at http://127.0.97.62:42319/ using document root <none> and password file <none>
I20260926 21:42:24.260603 22429 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:24.260732 22429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:24.261061 22429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:24.264132 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd"
format_stamp: "Formatted at 2026-09-26 21:42:24 on dist-test-slave-lhhq"
server_key: "9fc10d98bb3f8683837ad14e5d0e1d1c"
server_key_iv: "628f06f667282e7d3ceba59c2339218e"
server_key_version: "encryptionkey@0"
I20260926 21:42:24.264935 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd"
format_stamp: "Formatted at 2026-09-26 21:42:24 on dist-test-slave-lhhq"
server_key: "9fc10d98bb3f8683837ad14e5d0e1d1c"
server_key_iv: "628f06f667282e7d3ceba59c2339218e"
server_key_version: "encryptionkey@0"
I20260926 21:42:24.270505 22429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:42:24.274253 22444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:24.275715 22429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 21:42:24.275914 22429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd"
format_stamp: "Formatted at 2026-09-26 21:42:24 on dist-test-slave-lhhq"
server_key: "9fc10d98bb3f8683837ad14e5d0e1d1c"
server_key_iv: "628f06f667282e7d3ceba59c2339218e"
server_key_version: "encryptionkey@0"
I20260926 21:42:24.276129 22429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:24.293676 22429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:24.298377 22429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:24.298695 22429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:24.316938 22429 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38683
I20260926 21:42:24.316962 22495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38683 every 8 connection(s)
I20260926 21:42:24.318374 22429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:24.321509 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 22429
I20260926 21:42:24.321704 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:24.322280 388 external_mini_cluster.cc:1428] Setting key b5eb27b29115aca9a950fb6477243736
I20260926 21:42:24.322492 22496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:24.337590 22496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd: Bootstrap starting.
Sep 26 21:42:24 dist-test-slave-lhhq krb5kdc[22416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:42:24.341058 22496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:24.342217 22496 log.cc:824] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:24.345104 22496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd: No bootstrap required, opened a new log
I20260926 21:42:24.346611 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:24.324152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45697 (local address 127.0.97.62:38683)
0926 21:42:24.324944 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:24.324956 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:24.324995 (+ 39us) server_negotiation.cc:407] Connection header received
0926 21:42:24.327354 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:24.327378 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:24.327892 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:24.328257 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:24.330165 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:24.331462 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:24.333374 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:24.334101 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:24.334246 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:24.339062 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:24.339070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:24.339084 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:24.339132 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:24.342517 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:24.343255 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:24.343262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:24.343268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:24.343431 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:24.343821 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:24.343825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:24.343827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:24.344362 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:24.344677 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:24.345114 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 21:42:24.345583 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":365,"threads_started":1}
I20260926 21:42:24.350673 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:24.353228 22496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38683 } }
I20260926 21:42:24.353783 22496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:24.353899 22496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4cbd98ba204e18b3401d3c92d2ccbd, State: Initialized, Role: FOLLOWER
I20260926 21:42:24.354499 22496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38683 } }
I20260926 21:42:24.354732 22496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:24.354892 22496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:24.355077 22496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:24.357841 22496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38683 } }
I20260926 21:42:24.358457 22496 leader_election.cc:302] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a4cbd98ba204e18b3401d3c92d2ccbd; no voters:
I20260926 21:42:24.359112 22496 leader_election.cc:288] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:24.359256 22506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:24.359885 22506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [term 1 LEADER]: Becoming Leader. State: Replica: 2a4cbd98ba204e18b3401d3c92d2ccbd, State: Running, Role: LEADER
I20260926 21:42:24.360555 22506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38683 } }
I20260926 21:42:24.363224 22496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:24.363991 22508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38683 } } }
I20260926 21:42:24.364403 22508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:24.364944 22507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a4cbd98ba204e18b3401d3c92d2ccbd. Latest consensus state: current_term: 1 leader_uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cbd98ba204e18b3401d3c92d2ccbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38683 } } }
I20260926 21:42:24.365293 22507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:24.374622 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:24.355107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47357 (local address 127.0.97.62:38683)
0926 21:42:24.355314 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:24.355320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:24.355855 (+ 535us) server_negotiation.cc:407] Connection header received
0926 21:42:24.355976 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:24.355983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:24.356093 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:24.356210 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:24.357661 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:24.358756 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:24.360732 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:24.361449 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:24.361563 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:24.365111 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:24.365118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:24.365122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:24.365173 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:24.369792 (+ 4619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:24.370940 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:24.370949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:24.370955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:24.371144 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:24.371784 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:24.371792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:24.371798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:24.372344 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:24.372660 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:24.373658 (+ 998us) server_negotiation.cc:299] Negotiation successful
0926 21:42:24.374011 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 21:42:24.377400 22511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:24.382400 22511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:24.391252 22511 catalog_manager.cc:1347] Generated new cluster ID: cb3a71e72c31495493afbc46b2fddb5a
I20260926 21:42:24.391397 22511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:24.442698 22511 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:24.444777 22511 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:24.462567 22511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a4cbd98ba204e18b3401d3c92d2ccbd: Generated new TSK 0
I20260926 21:42:24.463729 22511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:24.471024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22429
I20260926 21:42:24.472512 22491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:24.567346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 22429
2026-09-26T21:42:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:24 dist-test-slave-lhhq krb5kdc[22529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:24 dist-test-slave-lhhq krb5kdc[22529](info): set up 2 sockets
Sep 26 21:42:24 dist-test-slave-lhhq krb5kdc[22529](info): commencing operation
krb5kdc: starting...
W20260926 21:42:25.669567 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:25 dist-test-slave-lhhq krb5kdc[22529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:42:25.936132 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43787
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:26.262663 22542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:26.263006 22542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:26.263178 22542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:26.270356 22542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:26.270510 22542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:26.270612 22542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:26.270715 22542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:26.270795 22542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:26.280556 22542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:26.281971 22542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:26.283344 22542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:26.295295 22547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:26.295230 22550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:26.295295 22548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:26.296058 22542 server_base.cc:1034] running on GCE node
I20260926 21:42:26.296852 22542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:26.298228 22542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:26.299441 22542 hybrid_clock.cc:648] HybridClock initialized: now 1790458946299383 us; error 50 us; skew 500 ppm
I20260926 21:42:26.299782 22542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:26 dist-test-slave-lhhq krb5kdc[22529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:26.308969 22542 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:42:26.310465 22542 webserver.cc:460] Webserver started at http://127.0.97.62:34711/ using document root <none> and password file <none>
I20260926 21:42:26.311137 22542 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:26.311264 22542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:26.311578 22542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:26.314577 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "0784034308364185911603bdf1fbd04b"
format_stamp: "Formatted at 2026-09-26 21:42:26 on dist-test-slave-lhhq"
server_key: "9279057aa7ebf5d062067e144f4c6939"
server_key_iv: "1e851b4c8a93eabd40e327e1e35d144e"
server_key_version: "encryptionkey@0"
I20260926 21:42:26.315339 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "0784034308364185911603bdf1fbd04b"
format_stamp: "Formatted at 2026-09-26 21:42:26 on dist-test-slave-lhhq"
server_key: "9279057aa7ebf5d062067e144f4c6939"
server_key_iv: "1e851b4c8a93eabd40e327e1e35d144e"
server_key_version: "encryptionkey@0"
I20260926 21:42:26.320443 22542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 21:42:26.323858 22557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:26.325201 22542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:42:26.325378 22542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "0784034308364185911603bdf1fbd04b"
format_stamp: "Formatted at 2026-09-26 21:42:26 on dist-test-slave-lhhq"
server_key: "9279057aa7ebf5d062067e144f4c6939"
server_key_iv: "1e851b4c8a93eabd40e327e1e35d144e"
server_key_version: "encryptionkey@0"
I20260926 21:42:26.325567 22542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:26.347411 22542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:26.351890 22542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:26.352190 22542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:26.370460 22542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43787
I20260926 21:42:26.370489 22608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43787 every 8 connection(s)
I20260926 21:42:26.371845 22542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:26.375872 22609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:26.380438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 22542
I20260926 21:42:26.380638 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:26.381232 388 external_mini_cluster.cc:1428] Setting key b8532f508dc1dffa482c543e65664313
I20260926 21:42:26.389012 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b: Bootstrap starting.
I20260926 21:42:26.392050 22609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:26.393072 22609 log.cc:824] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:26.394289 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:26.383165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55847 (local address 127.0.97.62:43787)
0926 21:42:26.383981 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:26.383993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:26.384036 (+ 43us) server_negotiation.cc:407] Connection header received
0926 21:42:26.385641 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:26.385663 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:26.386086 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:26.386483 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:26.387078 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:26.388350 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:26.389865 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:26.390566 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:26.390779 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:26.391932 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:26.391940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:26.391954 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:26.392565 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:26.392645 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:26.392674 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:26.392808 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:26.393029 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 21:42:26.393325 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":372,"threads_started":1}
I20260926 21:42:26.395874 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b: No bootstrap required, opened a new log
I20260926 21:42:26.397934 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:26.405213 22609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784034308364185911603bdf1fbd04b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43787 } }
I20260926 21:42:26.405795 22609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:26.405906 22609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0784034308364185911603bdf1fbd04b, State: Initialized, Role: FOLLOWER
I20260926 21:42:26.406647 22609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784034308364185911603bdf1fbd04b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43787 } }
I20260926 21:42:26.406987 22609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:26.407152 22609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:26.407341 22609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:26.410076 22609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784034308364185911603bdf1fbd04b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43787 } }
I20260926 21:42:26.410759 22609 leader_election.cc:302] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0784034308364185911603bdf1fbd04b; no voters:
I20260926 21:42:26.410975 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:26.402205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44061 (local address 127.0.97.62:43787)
0926 21:42:26.402379 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:26.402384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:26.404476 (+ 2092us) server_negotiation.cc:407] Connection header received
0926 21:42:26.404801 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:26.404807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:26.404890 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:26.404998 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:26.405650 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:26.406746 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:26.408275 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:26.408978 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:26.409123 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:26.410240 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:26.410248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:26.410254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:26.410278 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:26.410322 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:26.410341 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:26.410435 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:26.410529 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 21:42:26.410619 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 21:42:26.411489 22609 leader_election.cc:288] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:26.411623 22619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:26.413169 22619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [term 1 LEADER]: Becoming Leader. State: Replica: 0784034308364185911603bdf1fbd04b, State: Running, Role: LEADER
I20260926 21:42:26.413757 22619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784034308364185911603bdf1fbd04b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43787 } }
I20260926 21:42:26.413880 22609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:26.416882 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0784034308364185911603bdf1fbd04b. Latest consensus state: current_term: 1 leader_uuid: "0784034308364185911603bdf1fbd04b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784034308364185911603bdf1fbd04b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43787 } } }
I20260926 21:42:26.417326 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:26.417059 22620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0784034308364185911603bdf1fbd04b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0784034308364185911603bdf1fbd04b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43787 } } }
I20260926 21:42:26.417452 22620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:26.419521 22625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:26.426117 22625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:26.436823 22625 catalog_manager.cc:1347] Generated new cluster ID: 62cc2554dfe84f7d9fd71d3509d56bf2
I20260926 21:42:26.436951 22625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:26.453269 22625 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:26.454720 22625 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:26.460763 22625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0784034308364185911603bdf1fbd04b: Generated new TSK 0
I20260926 21:42:26.461859 22625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:26.467947 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22542
W20260926 21:42:26.469122 22604 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55847
W20260926 21:42:26.469810 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:26.469956 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 22542
2026-09-26T21:42:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:26 dist-test-slave-lhhq krb5kdc[22640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:26 dist-test-slave-lhhq krb5kdc[22640](info): set up 2 sockets
Sep 26 21:42:26 dist-test-slave-lhhq krb5kdc[22640](info): commencing operation
krb5kdc: starting...
W20260926 21:42:27.585667 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:27 dist-test-slave-lhhq krb5kdc[22640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:42:27.848440 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39351
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:28.178841 22653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:28.179142 22653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:28.179311 22653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:28.186806 22653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:28.186941 22653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:28.187016 22653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:28.187119 22653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:28.187182 22653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:28.197469 22653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:28.198856 22653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:28.200148 22653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:28.212107 22659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:28.212203 22661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:28.212170 22658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:28.213245 22653 server_base.cc:1034] running on GCE node
I20260926 21:42:28.214059 22653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:28.215547 22653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:28.216773 22653 hybrid_clock.cc:648] HybridClock initialized: now 1790458948216723 us; error 48 us; skew 500 ppm
I20260926 21:42:28.217142 22653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:28 dist-test-slave-lhhq krb5kdc[22640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:28.220206 22653 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:42:28.221761 22653 webserver.cc:460] Webserver started at http://127.0.97.62:46071/ using document root <none> and password file <none>
I20260926 21:42:28.222463 22653 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:28.222602 22653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:28.222920 22653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:28.226029 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c"
format_stamp: "Formatted at 2026-09-26 21:42:28 on dist-test-slave-lhhq"
server_key: "b390fa0d62392e053538b67bf9dcddc3"
server_key_iv: "540294390d514c7d963b18994125e923"
server_key_version: "encryptionkey@0"
I20260926 21:42:28.226789 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c"
format_stamp: "Formatted at 2026-09-26 21:42:28 on dist-test-slave-lhhq"
server_key: "b390fa0d62392e053538b67bf9dcddc3"
server_key_iv: "540294390d514c7d963b18994125e923"
server_key_version: "encryptionkey@0"
I20260926 21:42:28.231940 22653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:42:28.235371 22668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:28.236635 22653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:42:28.236821 22653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c"
format_stamp: "Formatted at 2026-09-26 21:42:28 on dist-test-slave-lhhq"
server_key: "b390fa0d62392e053538b67bf9dcddc3"
server_key_iv: "540294390d514c7d963b18994125e923"
server_key_version: "encryptionkey@0"
I20260926 21:42:28.237020 22653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:28.242704 22653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:28.246806 22653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:28.247098 22653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:28.265429 22653 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39351
I20260926 21:42:28.265480 22719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39351 every 8 connection(s)
I20260926 21:42:28.266860 22653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:28.270534 22720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:28.272716 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 22653
I20260926 21:42:28.272898 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:28.273378 388 external_mini_cluster.cc:1428] Setting key 99bad0274813042f1f129c51d3f6f7e9
I20260926 21:42:28.281265 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:28.275283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36023 (local address 127.0.97.62:39351)
0926 21:42:28.276110 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:28.276121 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:28.276168 (+ 47us) server_negotiation.cc:407] Connection header received
0926 21:42:28.277902 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:28.277925 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:28.278255 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:28.278580 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:28.279671 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:28.279678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:28.279693 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:28.279707 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:28.279771 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:28.279795 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:28.279968 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:28.280073 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 21:42:28.280279 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":397,"threads_started":1}
I20260926 21:42:28.282411 22720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c: Bootstrap starting.
I20260926 21:42:28.284754 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:28.285638 22720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:28.286602 22720 log.cc:824] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:28.289750 22720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c: No bootstrap required, opened a new log
I20260926 21:42:28.291471 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:28.289061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39381 (local address 127.0.97.62:39351)
0926 21:42:28.289264 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:28.289270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:28.289662 (+ 392us) server_negotiation.cc:407] Connection header received
0926 21:42:28.289806 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:28.289813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:28.289867 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:28.289968 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:28.290838 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:28.290844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:28.290848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 21:42:28.290865 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:28.290908 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 21:42:28.290928 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:28.291059 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:28.291108 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 21:42:28.291160 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 21:42:28.297708 22720 raft_consensus.cc:348] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39351 } }
I20260926 21:42:28.298090 22720 raft_consensus.cc:374] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 21:42:28.298197 22720 raft_consensus.cc:729] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93cf1f968de4ee6b9f7e33b139e1d4c, State: Initialized, Role: FOLLOWER
I20260926 21:42:28.298852 22720 consensus_queue.cc:260] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39351 } }
I20260926 21:42:28.299136 22720 raft_consensus.cc:388] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 21:42:28.299288 22720 raft_consensus.cc:482] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 21:42:28.299443 22720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:28.301638 22720 raft_consensus.cc:504] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39351 } }
I20260926 21:42:28.302114 22720 leader_election.cc:302] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c93cf1f968de4ee6b9f7e33b139e1d4c; no voters:
I20260926 21:42:28.302744 22720 leader_election.cc:288] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 21:42:28.302886 22730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:28.303987 22730 raft_consensus.cc:686] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [term 1 LEADER]: Becoming Leader. State: Replica: c93cf1f968de4ee6b9f7e33b139e1d4c, State: Running, Role: LEADER
I20260926 21:42:28.304474 22730 consensus_queue.cc:237] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39351 } }
I20260926 21:42:28.304725 22720 sys_catalog.cc:564] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:28.307395 22731 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39351 } } }
I20260926 21:42:28.307845 22731 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:28.307386 22732 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c93cf1f968de4ee6b9f7e33b139e1d4c. Latest consensus state: current_term: 1 leader_uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c93cf1f968de4ee6b9f7e33b139e1d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39351 } } }
I20260926 21:42:28.308378 22732 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:28.310556 22739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:28.315115 22739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 21:42:28.330173 22739 catalog_manager.cc:1347] Generated new cluster ID: c836fc7abf84483ca484f17841f2adc2
I20260926 21:42:28.330309 22739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:28.351470 22739 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:28.352011 22739 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:28.362569 22739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c93cf1f968de4ee6b9f7e33b139e1d4c: Generated new TSK 0
I20260926 21:42:28.363647 22739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:28.381436 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22653
W20260926 21:42:28.382561 22715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36023
W20260926 21:42:28.383268 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22653: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 21:42:28.383422 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 22653
2026-09-26T21:42:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1914 ms)
[----------] 18 tests from AuthTokenIssuingTest (35204 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:28 dist-test-slave-lhhq krb5kdc[22751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:28 dist-test-slave-lhhq krb5kdc[22751](info): set up 2 sockets
Sep 26 21:42:28 dist-test-slave-lhhq krb5kdc[22751](info): commencing operation
krb5kdc: starting...
W20260926 21:42:29.498955 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:29 dist-test-slave-lhhq krb5kdc[22751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:42:29.767032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46837
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46837,127.0.97.61:43763,127.0.97.60:40231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:30.092891 22764 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:42:30.093235 22764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:30.093348 22764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:30.093489 22764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:30.100819 22764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:30.100911 22764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:30.101035 22764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:30.101117 22764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:30.101238 22764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:30.111179 22764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46837,127.0.97.61:43763,127.0.97.60:40231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:30.112504 22764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:30.113897 22764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:30.125228 22769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:30.125171 22772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:30.125270 22770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:30.126154 22764 server_base.cc:1034] running on GCE node
I20260926 21:42:30.126916 22764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:30.128271 22764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:30.129513 22764 hybrid_clock.cc:648] HybridClock initialized: now 1790458950129452 us; error 38 us; skew 500 ppm
I20260926 21:42:30.129837 22764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:30 dist-test-slave-lhhq krb5kdc[22751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:30.132961 22764 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:42:30.134524 22764 webserver.cc:460] Webserver started at http://127.0.97.62:39747/ using document root <none> and password file <none>
I20260926 21:42:30.135211 22764 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:30.135341 22764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:30.135648 22764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:30.138573 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "fbb9879483844b82afec6333cb4cf593"
format_stamp: "Formatted at 2026-09-26 21:42:30 on dist-test-slave-lhhq"
server_key: "20b4c9321fb5e332a33c63cd9e4005d2"
server_key_iv: "60f70b5cffc1e77594404d56a09764f1"
server_key_version: "encryptionkey@0"
I20260926 21:42:30.139333 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "fbb9879483844b82afec6333cb4cf593"
format_stamp: "Formatted at 2026-09-26 21:42:30 on dist-test-slave-lhhq"
server_key: "20b4c9321fb5e332a33c63cd9e4005d2"
server_key_iv: "60f70b5cffc1e77594404d56a09764f1"
server_key_version: "encryptionkey@0"
I20260926 21:42:30.144192 22764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 21:42:30.147526 22779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:30.148770 22764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:42:30.148939 22764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "fbb9879483844b82afec6333cb4cf593"
format_stamp: "Formatted at 2026-09-26 21:42:30 on dist-test-slave-lhhq"
server_key: "20b4c9321fb5e332a33c63cd9e4005d2"
server_key_iv: "60f70b5cffc1e77594404d56a09764f1"
server_key_version: "encryptionkey@0"
I20260926 21:42:30.149138 22764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:30.218125 22764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:30.222731 22764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:30.223006 22764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:30.241958 22764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46837
I20260926 21:42:30.241991 22830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46837 every 8 connection(s)
I20260926 21:42:30.243320 22764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:30.247164 22831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:30.252764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 22764
I20260926 21:42:30.253018 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:30.253844 388 external_mini_cluster.cc:1428] Setting key 0a9ee318359fc918891649e7b46a2ff8
I20260926 21:42:30.254614 22831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:30 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260926 21:42:30.282923 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.258773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.0.97.62:46837)
0926 21:42:30.259747 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:30.259765 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:30.260537 (+ 772us) server_negotiation.cc:407] Connection header received
0926 21:42:30.263668 (+ 3131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:30.263674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:30.263996 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:30.264226 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:30.265504 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.267468 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.268922 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.269933 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.270116 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:30.273862 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:30.273876 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:30.273895 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:30.273939 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:30.278318 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:30.279143 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:30.279156 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:30.279162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:30.279370 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:30.279845 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:30.279852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:30.279862 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:30.280622 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:30.280859 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:30.281536 (+ 677us) server_negotiation.cc:299] Negotiation successful
0926 21:42:30.281927 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":709,"thread_start_us":441,"threads_started":1}
I20260926 21:42:30.282976 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.258773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46837 (local address 127.0.0.1:34300)
0926 21:42:30.260327 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.260382 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:30.263255 (+ 2873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:30.264265 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:30.264270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:30.264895 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:30.265302 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:30.265321 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.267634 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.267638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:30.268781 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:30.268793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.270190 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.270196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:30.270414 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:30.271170 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:30.271197 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:30.273606 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:30.278497 (+ 4891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:30.278508 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:30.278525 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:30.278972 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:30.279504 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:30.279508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:30.279510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:30.279713 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:30.280786 (+ 1073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:30.280793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:30.281004 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 21:42:30.281322 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 21:42:30.281748 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":51,"spinlock_wait_cycles":28544,"thread_start_us":1049,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 21:42:30.285967 22831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:42:30.287115 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.286756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51224)
0926 21:42:30.286911 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.287011 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 21:42:30.287465 22783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
I20260926 21:42:30.288933 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.288621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51238)
0926 21:42:30.288744 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.288813 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 21:42:30.289233 22831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 21:42:30.314615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43763
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46837,127.0.97.61:43763,127.0.97.60:40231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:42:30.338573 22831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } attempt: 1
I20260926 21:42:30.339748 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.339318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51250)
0926 21:42:30.339502 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.339622 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 21:42:30.340746 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.340445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51256)
0926 21:42:30.340563 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.340625 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 21:42:30.340934 22831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
I20260926 21:42:30.409211 22831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } attempt: 2
I20260926 21:42:30.410454 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.409986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51258)
0926 21:42:30.410211 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.410322 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 21:42:30.411473 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.411131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51260)
0926 21:42:30.411287 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.411355 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 21:42:30.411734 22831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
I20260926 21:42:30.503003 22831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } attempt: 3
I20260926 21:42:30.504191 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.503774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51270)
0926 21:42:30.503970 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.504059 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 21:42:30.505347 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.504845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51280)
0926 21:42:30.504993 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.505094 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 21:42:30.505570 22831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
I20260926 21:42:30.648834 22831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } attempt: 4
I20260926 21:42:30.650063 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.649503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51292)
0926 21:42:30.649785 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.649897 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260926 21:42:30.650092 22838 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:42:30.650516 22838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:30.650646 22838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:30.650789 22838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 21:42:30.651213 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.650799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51294)
0926 21:42:30.650962 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.651058 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 21:42:30.651463 22831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43763: connect: Connection refused (error 111)
W20260926 21:42:30.659214 22838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:30.659318 22838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:30.659377 22838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:30.659430 22838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:30.659493 22838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:30.669868 22838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46837,127.0.97.61:43763,127.0.97.60:40231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:30.671151 22838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:30.672457 22838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:30.684481 22843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:30.684408 22846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:30.684446 22844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:30.685398 22838 server_base.cc:1034] running on GCE node
I20260926 21:42:30.686204 22838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:30.687580 22838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:30.688794 22838 hybrid_clock.cc:648] HybridClock initialized: now 1790458950688718 us; error 44 us; skew 500 ppm
I20260926 21:42:30.689077 22838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:30 dist-test-slave-lhhq krb5kdc[22751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:30.692296 22838 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 21:42:30.693853 22838 webserver.cc:460] Webserver started at http://127.0.97.61:44299/ using document root <none> and password file <none>
I20260926 21:42:30.694547 22838 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:30.694658 22838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:30.694991 22838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:30.698082 22838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/data/instance:
uuid: "ee958f6735664176996b6405abf3277a"
format_stamp: "Formatted at 2026-09-26 21:42:30 on dist-test-slave-lhhq"
server_key: "ab66ab2c3b85083bb75622ccd8e5141e"
server_key_iv: "2a14fbd72a6ab1d6ddb3de09b1b0983b"
server_key_version: "encryptionkey@0"
I20260926 21:42:30.698940 22838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/wal/instance:
uuid: "ee958f6735664176996b6405abf3277a"
format_stamp: "Formatted at 2026-09-26 21:42:30 on dist-test-slave-lhhq"
server_key: "ab66ab2c3b85083bb75622ccd8e5141e"
server_key_iv: "2a14fbd72a6ab1d6ddb3de09b1b0983b"
server_key_version: "encryptionkey@0"
I20260926 21:42:30.704111 22838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 21:42:30.707474 22853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:30.708808 22838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 21:42:30.708981 22838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/wal
uuid: "ee958f6735664176996b6405abf3277a"
format_stamp: "Formatted at 2026-09-26 21:42:30 on dist-test-slave-lhhq"
server_key: "ab66ab2c3b85083bb75622ccd8e5141e"
server_key_iv: "2a14fbd72a6ab1d6ddb3de09b1b0983b"
server_key_version: "encryptionkey@0"
I20260926 21:42:30.709230 22838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:30.723774 22838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:30.727838 22838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:30.728103 22838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:30.745735 22838 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43763
I20260926 21:42:30.745754 22904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43763 every 8 connection(s)
I20260926 21:42:30.747098 22838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:42:30.749020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 22838
I20260926 21:42:30.749258 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-1/wal/instance
I20260926 21:42:30.749708 388 external_mini_cluster.cc:1428] Setting key 814c810611af22119d7c08e6f2cf3e34
I20260926 21:42:30.750903 22905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:30.757827 22905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:30 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 21:42:30.779122 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.760427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.0.97.62:46837)
0926 21:42:30.760660 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:30.760668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:30.761898 (+ 1230us) server_negotiation.cc:407] Connection header received
0926 21:42:30.764008 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:30.764017 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:30.764111 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:30.764239 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:30.765778 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.766921 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.768787 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.769579 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.769748 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:30.773069 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:30.773075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:30.773078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:30.773111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:30.775828 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:30.776810 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:30.776815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:30.776817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:30.776937 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:30.777434 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:30.777437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:30.777439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:30.777714 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:30.777838 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:30.778734 (+ 896us) server_negotiation.cc:299] Negotiation successful
0926 21:42:30.778902 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:42:30.780685 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.760862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46837 (local address 127.0.0.1:34306)
0926 21:42:30.761721 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.761771 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:30.763690 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:30.764432 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:30.764439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:30.764967 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:30.765604 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:30.765623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.767089 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.767094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:30.768604 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:30.768623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.769794 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.769799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:30.770003 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:30.770685 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:30.770708 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:30.772864 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:30.775973 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:30.775986 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:30.776007 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:30.776672 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:30.777051 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:30.777060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:30.777066 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:30.777326 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:30.777856 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:30.777867 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:30.778348 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 21:42:30.778651 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 21:42:30.779207 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":377,"threads_started":1}
I20260926 21:42:30.782516 22905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:30 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:42:30.799563 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.783394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51298)
0926 21:42:30.783579 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.783604 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:30.783710 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:30.784820 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:30.784828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:30.785424 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:30.785622 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:30.785636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.787756 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.787760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:30.788587 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:30.788597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.789839 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.789844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:30.790022 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:30.790634 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:30.790651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:30.792606 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:30.797056 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:30.797061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:30.797064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:30.797486 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:30.797927 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:30.797930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:30.797932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:30.798059 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:30.798855 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:30.798859 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:30.798974 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 21:42:30.799085 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 21:42:30.799289 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":35}
I20260926 21:42:30.800062 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.783527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.0.97.61:43763)
0926 21:42:30.784284 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:30.784294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:30.784327 (+ 33us) server_negotiation.cc:407] Connection header received
0926 21:42:30.784383 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:30.784394 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:30.784637 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:30.784795 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:30.785794 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.787602 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.788721 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.789688 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.789836 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:30.792843 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:30.792859 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:30.792880 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:30.792921 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:30.796894 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:30.797616 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:30.797629 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:30.797636 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:30.797789 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:30.798172 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:30.798180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:30.798185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:30.798706 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:30.798937 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:30.799288 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 21:42:30.799602 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":558,"thread_start_us":418,"threads_started":1}
I20260926 21:42:30.802695 22905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:42:30.803679 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.803284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56546)
0926 21:42:30.803436 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.803543 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 21:42:30.803959 22857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
I20260926 21:42:30.805591 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.805258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56556)
0926 21:42:30.805379 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.805455 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 21:42:30.805861 22905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40231: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 21:42:30.812853 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40231
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46837,127.0.97.61:43763,127.0.97.60:40231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:42:30.855166 22905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } attempt: 1
I20260926 21:42:30.856288 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.855871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56560)
0926 21:42:30.856051 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.856139 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 21:42:30.857357 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.856992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56568)
0926 21:42:30.857127 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.857239 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 21:42:30.857599 22905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40231: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
I20260926 21:42:30.925808 22905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } attempt: 2
I20260926 21:42:30.926971 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.926502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56580)
0926 21:42:30.926706 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.926818 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 21:42:30.928009 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.927703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56590)
0926 21:42:30.927844 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.927906 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 21:42:30.928267 22905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40231: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
I20260926 21:42:30.950718 22831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } attempt: 5
Sep 26 21:42:30 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:42:30.964998 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.951703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.0.97.61:43763)
0926 21:42:30.951930 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:30.951939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:30.951967 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:42:30.952044 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:30.952052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:30.952147 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:30.952277 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:30.953159 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.954357 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.955513 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.956277 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.956420 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:30.959780 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:30.959786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:30.959789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:30.959825 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:30.962522 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:30.963280 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:30.963284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:30.963286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:30.963414 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:30.963793 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:30.963798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:30.963801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:30.964063 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:30.964170 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:30.964618 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 21:42:30.964779 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 21:42:30.965433 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.951513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51314)
0926 21:42:30.951733 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.951756 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:30.951894 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:30.952342 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:30.952345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:30.952818 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:30.953001 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:30.953009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.954496 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.954500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:30.955364 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:30.955374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:30.956424 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:30.956428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:30.956560 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:30.957216 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:30.957234 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:30.959589 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:30.962657 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:30.962662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:30.962666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:30.963174 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:30.963522 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:30.963527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:30.963530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:30.963697 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:30.964175 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:30.964179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:30.964376 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 21:42:30.964543 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 21:42:30.964901 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 21:42:30.966500 22831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:42:30.967480 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.967114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56598)
0926 21:42:30.967263 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.967343 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 21:42:30.968525 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:30.968109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56606)
0926 21:42:30.968282 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:30.968394 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 21:42:30.968755 22831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40231: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
I20260926 21:42:31.019487 22905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } attempt: 3
I20260926 21:42:31.019944 22831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } attempt: 1
I20260926 21:42:31.020792 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.020211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56618)
0926 21:42:31.020501 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.020605 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 21:42:31.021263 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.020535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56624)
0926 21:42:31.020738 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.020839 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 21:42:31.021981 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.021427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56634)
0926 21:42:31.021745 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.021839 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
W20260926 21:42:31.022199 22905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40231: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
I20260926 21:42:31.022279 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.021928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56642)
0926 21:42:31.022069 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.022145 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 21:42:31.022502 22831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40231: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
I20260926 21:42:31.090745 22831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } attempt: 2
I20260926 21:42:31.091933 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.091444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56656)
0926 21:42:31.091681 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.091767 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 21:42:31.092904 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.092583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56666)
0926 21:42:31.092729 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.092804 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 21:42:31.093147 22831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40231: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
W20260926 21:42:31.154476 22912 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:42:31.154793 22912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:31.154875 22912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:31.154958 22912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:31.162174 22912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:31.162271 22912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:31.162328 22912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:31.162381 22912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:31.162442 22912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:31.165426 22905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } attempt: 4
I20260926 21:42:31.166579 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.166107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56672)
0926 21:42:31.166342 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.166427 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 21:42:31.167500 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.167191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56680)
0926 21:42:31.167342 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.167407 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 21:42:31.167686 22905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40231: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
I20260926 21:42:31.172886 22912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46837,127.0.97.61:43763,127.0.97.60:40231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:31.174250 22912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:31.175570 22912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:31.187469 22917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:31.187381 22920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:31.187438 22918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:31.190884 22912 server_base.cc:1034] running on GCE node
I20260926 21:42:31.191738 22912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:31.193286 22912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:31.194520 22912 hybrid_clock.cc:648] HybridClock initialized: now 1790458951194465 us; error 41 us; skew 500 ppm
I20260926 21:42:31.194860 22912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:31 dist-test-slave-lhhq krb5kdc[22751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:31.198207 22912 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 21:42:31.199399 22831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } attempt: 3
I20260926 21:42:31.199823 22912 webserver.cc:460] Webserver started at http://127.0.97.60:33323/ using document root <none> and password file <none>
I20260926 21:42:31.200501 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.200018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56686)
0926 21:42:31.200240 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.200346 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 21:42:31.200685 22912 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:31.200798 22912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:31.201188 22912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:31.201529 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.201144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56688)
0926 21:42:31.201356 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.201432 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 21:42:31.201812 22831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40231: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40231: connect: Connection refused (error 111)
I20260926 21:42:31.204383 22912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/data/instance:
uuid: "7979fbacb4d14837b61e46a33a1d15b8"
format_stamp: "Formatted at 2026-09-26 21:42:31 on dist-test-slave-lhhq"
server_key: "9d26575198a9bc49daaea573d84e1025"
server_key_iv: "48c633235882206154b3f54d391ccd1d"
server_key_version: "encryptionkey@0"
I20260926 21:42:31.205169 22912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/wal/instance:
uuid: "7979fbacb4d14837b61e46a33a1d15b8"
format_stamp: "Formatted at 2026-09-26 21:42:31 on dist-test-slave-lhhq"
server_key: "9d26575198a9bc49daaea573d84e1025"
server_key_iv: "48c633235882206154b3f54d391ccd1d"
server_key_version: "encryptionkey@0"
I20260926 21:42:31.210366 22912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 21:42:31.213894 22927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:31.215276 22912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:42:31.215461 22912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/wal
uuid: "7979fbacb4d14837b61e46a33a1d15b8"
format_stamp: "Formatted at 2026-09-26 21:42:31 on dist-test-slave-lhhq"
server_key: "9d26575198a9bc49daaea573d84e1025"
server_key_iv: "48c633235882206154b3f54d391ccd1d"
server_key_version: "encryptionkey@0"
I20260926 21:42:31.215665 22912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:31.245992 22912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:31.250411 22912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:31.250792 22912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:31.269791 22912 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40231
I20260926 21:42:31.269814 22978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40231 every 8 connection(s)
I20260926 21:42:31.271198 22912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:42:31.271708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 22912
I20260926 21:42:31.271895 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790458593267894-388-0/minicluster-data/master-2/wal/instance
I20260926 21:42:31.272500 388 external_mini_cluster.cc:1428] Setting key b70c7d7bb2839663f0848f59f2643a0f
I20260926 21:42:31.275990 22979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:31.284960 22979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:31 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:42:31 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:42:31.296620 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.274436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.0.97.60:40231)
0926 21:42:31.275213 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.275230 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.275277 (+ 47us) server_negotiation.cc:407] Connection header received
0926 21:42:31.277674 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.277703 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.278185 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.278633 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.280218 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.282327 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.283626 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.284645 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.284838 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.289536 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.289544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.289554 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.289595 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.292564 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.293272 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.293285 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.293292 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.293459 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.293826 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.293839 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.293841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.294342 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.294677 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.295145 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.295539 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":312,"threads_started":1}
I20260926 21:42:31.302984 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.287647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.97.62:46837)
0926 21:42:31.288933 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.288938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.288953 (+ 15us) server_negotiation.cc:407] Connection header received
0926 21:42:31.289002 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.289006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.289062 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.289180 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.290587 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.291528 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.292940 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.293402 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.293540 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.296631 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.296636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.296639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.296671 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.300131 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.300917 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.300938 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.300943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.301136 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.301539 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.301544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.301549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.301915 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.302061 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.302510 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.302673 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1178,"thread_start_us":1092,"threads_started":1}
I20260926 21:42:31.303117 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.287573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46837 (local address 127.0.0.1:34314)
0926 21:42:31.288310 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.288356 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:31.288494 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:31.289270 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:31.289277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:31.290030 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:31.290416 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.290433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.291682 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.291688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.292802 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.292814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.293587 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.293593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.293756 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.294343 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:31.294365 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:31.296448 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:31.300250 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.300264 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.300272 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.300810 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.301298 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.301301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.301303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.301431 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.302048 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:31.302053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:31.302196 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 21:42:31.302376 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 21:42:31.302712 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":341,"threads_started":1}
I20260926 21:42:31.304489 22979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:31 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 21:42:31 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:42:31.321635 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.299959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.0.97.62:46837)
0926 21:42:31.300691 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.300697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.300720 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:42:31.300769 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.300774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.300858 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.300962 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.302417 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.303685 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.305088 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.305629 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.305762 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.309720 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.309725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.309728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.309756 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.318409 (+ 8653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.319200 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.319205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.319208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.319358 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.319787 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.319793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.319799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.320198 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.320406 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.320986 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.321150 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":368,"threads_started":1}
I20260926 21:42:31.324810 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.305228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43763 (local address 127.0.0.1:51320)
0926 21:42:31.305387 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.305405 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:31.305494 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:31.305787 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:31.305799 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:31.306131 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:31.306260 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.306267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.307191 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.307197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.308033 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.308043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.308756 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.308762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.308947 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.316281 (+ 7334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:31.316311 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:31.318621 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:31.322373 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.322378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.322382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.322718 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.323133 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.323138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.323142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.323320 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.323848 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:31.323853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:31.324012 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 21:42:31.324148 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 21:42:31.324454 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 21:42:31.324856 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.305277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.0.97.61:43763)
0926 21:42:31.305470 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.305475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.305494 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:42:31.305599 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.305604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.305674 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.305772 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.306401 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.307057 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.308187 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.308599 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.308724 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.318864 (+ 10140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.318870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.318873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.318925 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.322241 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.322850 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.322856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.322859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.322997 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.323443 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.323448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.323452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.323729 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.323889 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.324252 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.324452 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 21:42:31.326017 22979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:31 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:42:31 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:42:31.339134 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.326600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56700)
0926 21:42:31.326785 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.326803 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:31.326886 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:31.327136 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:31.327140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:31.327497 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:31.327645 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.327655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.328765 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.328789 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.329890 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.329903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.330558 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.330563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.330698 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.331207 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:31.331223 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:31.333256 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:31.336746 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.336749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.336752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.337116 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.337525 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.337531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.337534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.337687 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.338239 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:31.338244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:31.338404 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 21:42:31.338577 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 21:42:31.338796 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 21:42:31.339475 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.326767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.0.97.60:40231)
0926 21:42:31.326901 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.326905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.326933 (+ 28us) server_negotiation.cc:407] Connection header received
0926 21:42:31.326975 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.326979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.327034 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.327122 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.327775 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.328620 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.330020 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.330405 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.330536 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.333405 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.333410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.333413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.333439 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.336573 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.337248 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.337256 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.337262 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.337419 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.337788 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.337792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.337795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.338123 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.338256 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.338679 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.339007 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":46}
I20260926 21:42:31.340029 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.323242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.0.97.61:43763)
0926 21:42:31.325066 (+ 1824us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.325071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.325087 (+ 16us) server_negotiation.cc:407] Connection header received
0926 21:42:31.325129 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.325134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.325247 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.325355 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.326803 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.327532 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.329218 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.329631 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.329724 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.334134 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.334144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.334149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.334198 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.337226 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.337968 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.337973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.337977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.338120 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.338524 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.338527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.338529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.338883 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.339148 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.339492 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.339695 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1723,"thread_start_us":2114,"threads_started":1}
I20260926 21:42:31.344511 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:31.344727 22979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8: Bootstrap starting.
I20260926 21:42:31.347751 22979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:31.348836 22979 log.cc:824] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:31.347096 22806 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fbb9879483844b82afec6333cb4cf593"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34318
I20260926 21:42:31.351519 22979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8: No bootstrap required, opened a new log
I20260926 21:42:31.358582 22979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } }
I20260926 21:42:31.358914 22979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7979fbacb4d14837b61e46a33a1d15b8, State: Initialized, Role: FOLLOWER
I20260926 21:42:31.359351 22979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } }
I20260926 21:42:31.360323 22988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.360720 22988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:31.361285 22979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:42:31.370728 22999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:42:31.370867 22999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:42:31.379030 22831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } attempt: 4
I20260926 21:42:31.368702 22912 master_runner.cc:186] Error getting master registration for 127.0.97.62:46837: OK, instance_id { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" instance_seqno: 1790458950236451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:42:31.380025 22912 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 21:42:31 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:42:31.393926 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.379765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56714)
0926 21:42:31.380071 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.380093 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:31.380219 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:31.380638 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:31.380643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:31.381148 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:31.381398 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.381414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.382412 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.382421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.383897 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.383912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.384688 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.384694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.384900 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.385554 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:31.385572 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:31.387674 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:31.391496 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.391501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.391505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.391864 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.392208 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.392215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.392218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.392352 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.392907 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:31.392915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:31.393078 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 21:42:31.393247 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 21:42:31.393533 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260926 21:42:31.393921 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.380110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.0.97.60:40231)
0926 21:42:31.380300 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.380310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.380333 (+ 23us) server_negotiation.cc:407] Connection header received
0926 21:42:31.380393 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.380401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.380480 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.380598 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.381534 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.382264 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.384037 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.384511 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.384639 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.387854 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.387859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.387863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.387897 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.391361 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.391972 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.391980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.391983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.392105 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.392458 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.392463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.392466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.392792 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.392955 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.393340 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.393540 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 21:42:31.400255 22831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593: Bootstrap starting.
I20260926 21:42:31.404565 22831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:31.405925 22831 log.cc:824] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:31.409518 22831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593: No bootstrap required, opened a new log
I20260926 21:42:31.417145 22831 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } }
I20260926 21:42:31.417595 22831 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb9879483844b82afec6333cb4cf593, State: Initialized, Role: FOLLOWER
I20260926 21:42:31.418104 22831 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } }
I20260926 21:42:31.419250 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.419723 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:31.420584 22831 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:31.420646 22806 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 21:42:31.420902 22806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:31.424667 22806 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } }
I20260926 21:42:31.427785 22954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbb9879483844b82afec6333cb4cf593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7979fbacb4d14837b61e46a33a1d15b8"
I20260926 21:42:31.428344 22954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:31.432123 22954 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb9879483844b82afec6333cb4cf593 in term 1.
I20260926 21:42:31.432914 22783 leader_election.cc:302] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7979fbacb4d14837b61e46a33a1d15b8, fbb9879483844b82afec6333cb4cf593; no voters:
I20260926 21:42:31.433604 22806 leader_election.cc:288] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [CANDIDATE]: Term 1 election: Requested vote from peers ee958f6735664176996b6405abf3277a (127.0.97.61:43763), 7979fbacb4d14837b61e46a33a1d15b8 (127.0.97.60:40231)
I20260926 21:42:31.433643 23002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:31.440110 23002 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [term 1 LEADER]: Becoming Leader. State: Replica: fbb9879483844b82afec6333cb4cf593, State: Running, Role: LEADER
I20260926 21:42:31.440793 23002 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } }
I20260926 21:42:31.451875 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb9879483844b82afec6333cb4cf593. Latest consensus state: current_term: 1 leader_uuid: "fbb9879483844b82afec6333cb4cf593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.452456 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:31.457103 23014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:31.466874 22905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } attempt: 5
I20260926 21:42:31.471061 23014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 21:42:31.475858 22785 tablet.cc:2369] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:42:31.502987 22933 tablet.cc:2369] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:42:31.510918 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.462478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.0.97.61:43763)
0926 21:42:31.462696 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.462703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.477172 (+ 14469us) server_negotiation.cc:407] Connection header received
0926 21:42:31.477491 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.477498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.477584 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.477653 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.485343 (+ 7690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.486476 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.492137 (+ 5661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.492869 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.492971 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.501061 (+ 8090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.501069 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.501074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.501116 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.504768 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.505675 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.505681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.505685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.505858 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.506311 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.506316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.506320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.506677 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.506928 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.510301 (+ 3373us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.510547 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 21:42:31.521476 22954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [term 1 FOLLOWER]: Refusing update from remote peer fbb9879483844b82afec6333cb4cf593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:42:31.526135 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.467538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.0.97.60:40231)
0926 21:42:31.467775 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.467783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.480092 (+ 12309us) server_negotiation.cc:407] Connection header received
0926 21:42:31.480356 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.480361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.480444 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.480515 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.488797 (+ 8282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.490767 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.502037 (+ 11270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.503009 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.503142 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.512240 (+ 9098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.512246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.512251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.512300 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.516897 (+ 4597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.523253 (+ 6356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.523268 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.523272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.523444 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.523891 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.523896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.523900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.524279 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.524537 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.525299 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.525737 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 21:42:31.523558 23009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 21:42:31.536783 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.461367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.0.97.62:46837)
0926 21:42:31.461659 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.461669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.466724 (+ 5055us) server_negotiation.cc:407] Connection header received
0926 21:42:31.473579 (+ 6855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.473587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.473693 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.473769 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.483121 (+ 9352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.484294 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.490131 (+ 5837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.490858 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.490965 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.509228 (+ 18263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.509239 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.509246 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.509307 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.518540 (+ 9233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.533318 (+ 14778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.533324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.533327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.533519 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.534907 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.534914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.534917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.535290 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.535550 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.536136 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.536417 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 21:42:31 dist-test-slave-lhhq krb5kdc[22751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:42:31.533622 22764 master_runner.cc:186] Error getting master registration for 127.0.97.61:43763: OK, instance_id { permanent_uuid: "ee958f6735664176996b6405abf3277a" instance_seqno: 1790458950740593 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:42:31.544332 22764 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:42:31.548391 22988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb9879483844b82afec6333cb4cf593. Latest consensus state: current_term: 1 leader_uuid: "fbb9879483844b82afec6333cb4cf593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.548704 22988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:31.550424 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.467490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40231 (local address 127.0.0.1:56732)
0926 21:42:31.467731 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:31.467749 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:31.467876 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:31.497292 (+ 29416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:31.497301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:31.497862 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:31.498085 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.498102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.522858 (+ 24756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.522866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.524382 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:31.524402 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.537978 (+ 13576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.537985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:31.538218 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.539284 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:31.539309 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:31.542200 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:31.547460 (+ 5260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.547465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.547469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.548014 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.548507 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:31.548512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:31.548516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:31.548694 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:31.549406 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:31.549411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:31.549580 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 21:42:31.549747 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 21:42:31.550047 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260926 21:42:31.550472 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.468148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.0.97.60:40231)
0926 21:42:31.495260 (+ 27112us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.495270 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.495477 (+ 207us) server_negotiation.cc:407] Connection header received
0926 21:42:31.495901 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.495910 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.496712 (+ 802us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.497493 (+ 781us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.499698 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.522710 (+ 23012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.533672 (+ 10962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.534386 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.542932 (+ 8546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.543201 (+ 269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:31.543208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:31.543213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:31.543246 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:31.547303 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.548177 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.548181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.548185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.548367 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:31.548852 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:31.548858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:31.548864 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:31.549263 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:31.549402 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.549855 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.550078 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25833,"thread_start_us":20679,"threads_started":1}
I20260926 21:42:31.559250 22905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a: Bootstrap starting.
I20260926 21:42:31.561652 23024 mvcc.cc:205] Tried to move back new op lower bound from 7333719865416896512 to 7333719865108271104. Current Snapshot: MvccSnapshot[applied={T|T < 7333719865416896512}]
I20260926 21:42:31.563634 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbb9879483844b82afec6333cb4cf593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.563998 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:31.565375 22988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbb9879483844b82afec6333cb4cf593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.565613 22988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:31.565882 23014 catalog_manager.cc:1347] Generated new cluster ID: 604b684d65dc42cabd58e4667a462ad4
I20260926 21:42:31.565981 23014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:31.577502 22905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:31.579003 22905 log.cc:824] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:31.587430 22905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a: No bootstrap required, opened a new log
I20260926 21:42:31.614153 22905 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } }
I20260926 21:42:31.614753 22905 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee958f6735664176996b6405abf3277a, State: Initialized, Role: FOLLOWER
I20260926 21:42:31.616704 22905 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } }
I20260926 21:42:31.618168 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.619050 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:31.619314 23014 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:31.619704 22905 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:31.619752 22880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbb9879483844b82afec6333cb4cf593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ee958f6735664176996b6405abf3277a"
I20260926 21:42:31.620225 22880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:31.621287 23014 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:31.623883 22880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb9879483844b82afec6333cb4cf593 in term 1.
I20260926 21:42:31.624934 22879 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [term 1 FOLLOWER]: Refusing update from remote peer fbb9879483844b82afec6333cb4cf593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:42:31.626983 23009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:42:31.657392 23014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593: Generated new TSK 0
I20260926 21:42:31.658725 23014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:31.683987 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb9879483844b82afec6333cb4cf593. Latest consensus state: current_term: 1 leader_uuid: "fbb9879483844b82afec6333cb4cf593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.683995 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbb9879483844b82afec6333cb4cf593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.684402 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:31.684453 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:31.685877 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbb9879483844b82afec6333cb4cf593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb9879483844b82afec6333cb4cf593" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46837 } } peers { permanent_uuid: "ee958f6735664176996b6405abf3277a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43763 } } peers { permanent_uuid: "7979fbacb4d14837b61e46a33a1d15b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40231 } } }
I20260926 21:42:31.686199 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb9879483844b82afec6333cb4cf593 [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:31.715502 23049 catalog_manager.cc:1259] Loaded cluster ID: 604b684d65dc42cabd58e4667a462ad4
I20260926 21:42:31.715683 23049 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a: loading cluster ID for follower catalog manager: success
I20260926 21:42:31.734061 23049 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a: acquiring CA information for follower catalog manager: success
I20260926 21:42:31.736935 23049 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee958f6735664176996b6405abf3277a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 21:42:31.800205 22985 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:31.787064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.0.97.62:46837)
0926 21:42:31.787274 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.787281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.796383 (+ 9102us) server_negotiation.cc:407] Connection header received
0926 21:42:31.798327 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.798335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.798426 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.798507 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.799693 (+ 1186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34330: BlockingRecv error: recv got EOF from 127.0.0.1:34330 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 21:42:31.805517 22909 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:31.790870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.97.61:43763)
0926 21:42:31.791097 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.791104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.802573 (+ 11469us) server_negotiation.cc:407] Connection header received
0926 21:42:31.804019 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.804026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.804115 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.804206 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.805254 (+ 1048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51348: BlockingRecv error: recv got EOF from 127.0.0.1:51348 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 21:42:31.807575 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:31.796943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.0.97.60:40231)
0926 21:42:31.797132 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.797139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.803180 (+ 6041us) server_negotiation.cc:407] Connection header received
0926 21:42:31.806052 (+ 2872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.806058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.806186 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.806269 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.807348 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56742: BlockingRecv error: recv got EOF from 127.0.0.1:56742 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:42:31.861413 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.845643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.97.61:43763)
0926 21:42:31.845846 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.845853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.853353 (+ 7500us) server_negotiation.cc:407] Connection header received
0926 21:42:31.853709 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.853718 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.853955 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.854104 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:31.854896 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.856038 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.858190 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.858885 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.858989 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.859764 (+ 775us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:31.860534 (+ 770us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:31.860656 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.860921 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.861022 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 21:42:31.908075 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:31.894854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.0.97.60:40231)
0926 21:42:31.895059 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.895067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.905939 (+ 10872us) server_negotiation.cc:407] Connection header received
0926 21:42:31.906198 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.906204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.906436 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.906565 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:31.907757 (+ 1192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56746: BlockingRecv error: recv got EOF from 127.0.0.1:56746 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 21:42:31.957988 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:31.943272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.0.97.61:43763)
0926 21:42:31.943514 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.943521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:31.950503 (+ 6982us) server_negotiation.cc:407] Connection header received
0926 21:42:31.951032 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:31.951040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:31.951164 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:31.951238 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:31.952070 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.953380 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.955565 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:31.956278 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:31.956381 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:31.957013 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:31.957236 (+ 223us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:31.957310 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:31.957565 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 21:42:31.957657 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 21:42:32.003279 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:31.993753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.0.97.60:40231)
0926 21:42:31.993992 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:31.993999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.001285 (+ 7286us) server_negotiation.cc:407] Connection header received
0926 21:42:32.001644 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.001651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.001753 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.001830 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:32.003017 (+ 1187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56752: BlockingRecv error: recv got EOF from 127.0.0.1:56752 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 21:42:32.064155 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.049328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.0.97.61:43763)
0926 21:42:32.049511 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.049518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.056910 (+ 7392us) server_negotiation.cc:407] Connection header received
0926 21:42:32.057468 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.057475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.057564 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.057628 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.058320 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.059569 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.061797 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.062501 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.062599 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.063247 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.063385 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.063443 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.063675 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.063783 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 21:42:32.095155 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:32.090970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.0.97.60:40231)
0926 21:42:32.091210 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.091220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.093791 (+ 2571us) server_negotiation.cc:407] Connection header received
0926 21:42:32.094005 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.094015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.094094 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.094207 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:32.094962 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56768: BlockingRecv error: recv got EOF from 127.0.0.1:56768 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 21:42:32.108596 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.103011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.0.97.61:43763)
0926 21:42:32.103329 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.103339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.103869 (+ 530us) server_negotiation.cc:407] Connection header received
0926 21:42:32.104089 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.104094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.104181 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.104282 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.104702 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.105533 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.106847 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.107303 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.107436 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.107854 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.107958 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.108054 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.108189 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.108267 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
W20260926 21:42:32.116796 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:32.114716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.0.97.60:40231)
0926 21:42:32.114948 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.114955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.115495 (+ 540us) server_negotiation.cc:407] Connection header received
0926 21:42:32.115687 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.115693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.115775 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.115877 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:32.116598 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56770: BlockingRecv error: recv got EOF from 127.0.0.1:56770 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 21:42:32.134090 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.128452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.97.61:43763)
0926 21:42:32.128642 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.128649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.129378 (+ 729us) server_negotiation.cc:407] Connection header received
0926 21:42:32.129617 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.129623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.129700 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.129815 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.130199 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.130954 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.132241 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.132694 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.132838 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.133226 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.133355 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.133478 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.133658 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.133741 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 21:42:32.145453 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:32.143212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.0.97.60:40231)
0926 21:42:32.143451 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.143461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.144003 (+ 542us) server_negotiation.cc:407] Connection header received
0926 21:42:32.144229 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.144234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.144320 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.144426 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:32.145159 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56786: BlockingRecv error: recv got EOF from 127.0.0.1:56786 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 21:42:32.170176 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.164666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.97.61:43763)
0926 21:42:32.164882 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.164892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.165446 (+ 554us) server_negotiation.cc:407] Connection header received
0926 21:42:32.165669 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.165674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.165740 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.165844 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.166229 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.167034 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.168362 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.168837 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.168979 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.169421 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.169521 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.169623 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.169761 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.169856 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 21:42:32.178484 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:32.176379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.0.97.60:40231)
0926 21:42:32.176621 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.176629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.177129 (+ 500us) server_negotiation.cc:407] Connection header received
0926 21:42:32.177338 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.177343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.177436 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.177558 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:32.178293 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56792: BlockingRecv error: recv got EOF from 127.0.0.1:56792 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 21:42:32.220703 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.214440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.0.97.61:43763)
0926 21:42:32.214673 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.214680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.214987 (+ 307us) server_negotiation.cc:407] Connection header received
0926 21:42:32.215153 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.215159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.215274 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.215404 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.215853 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.217035 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.218481 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.219133 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.219282 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.219759 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.219896 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.219991 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.220212 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.220280 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 21:42:32.230028 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:32.227535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.0.97.60:40231)
0926 21:42:32.227792 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.227799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.228652 (+ 853us) server_negotiation.cc:407] Connection header received
0926 21:42:32.228870 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.228874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.228955 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.229056 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:32.229778 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56798: BlockingRecv error: recv got EOF from 127.0.0.1:56798 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 21:42:32.302928 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.297862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.0.97.61:43763)
0926 21:42:32.298106 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.298113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.298404 (+ 291us) server_negotiation.cc:407] Connection header received
0926 21:42:32.298566 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.298571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.298655 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.298742 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.299096 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.299852 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.301333 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.301785 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.301899 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.302243 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.302346 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.302424 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.302536 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.302599 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 21:42:32.310474 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:32.308655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.0.97.60:40231)
0926 21:42:32.308893 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.308899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.309117 (+ 218us) server_negotiation.cc:407] Connection header received
0926 21:42:32.309407 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.309411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.309483 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.309566 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:32.310268 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56810: BlockingRecv error: recv got EOF from 127.0.0.1:56810 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 21:42:32.373008 22999 catalog_manager.cc:1259] Loaded cluster ID: 604b684d65dc42cabd58e4667a462ad4
I20260926 21:42:32.373100 22999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8: loading cluster ID for follower catalog manager: success
I20260926 21:42:32.375598 22999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8: acquiring CA information for follower catalog manager: success
I20260926 21:42:32.377458 22999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7979fbacb4d14837b61e46a33a1d15b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:42:32.448077 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.442913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.0.97.61:43763)
0926 21:42:32.443128 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.443136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.443475 (+ 339us) server_negotiation.cc:407] Connection header received
0926 21:42:32.443721 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.443725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.443824 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.443925 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.444313 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.445018 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.446406 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.446821 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.446968 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.447350 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.447451 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.447547 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.447688 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.447771 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 21:42:32.459227 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.453924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.0.97.60:40231)
0926 21:42:32.454149 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.454156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.454432 (+ 276us) server_negotiation.cc:407] Connection header received
0926 21:42:32.454578 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.454582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.454731 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.454839 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.455179 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.455898 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.457144 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.457605 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.457699 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.458151 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.458656 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.458765 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.458839 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.458904 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 21:42:32.474107 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.465531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.0.97.61:43763)
0926 21:42:32.465811 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.465820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.467515 (+ 1695us) server_negotiation.cc:407] Connection header received
0926 21:42:32.467757 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.467766 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.467895 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.468017 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.468986 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.470051 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.472025 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.472707 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.472865 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.473367 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.473469 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.473569 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.473736 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.473818 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 21:42:32.475841 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.466541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.0.97.60:40231)
0926 21:42:32.466794 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.466803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.469871 (+ 3068us) server_negotiation.cc:407] Connection header received
0926 21:42:32.470068 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.470074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.470173 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.470271 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.471009 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.472066 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.474009 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.474547 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.474688 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.475109 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.475237 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.475334 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.475469 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.475548 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 21:42:32.477057 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:32.465413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.0.97.62:46837)
0926 21:42:32.470425 (+ 5012us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:32.470431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:32.470450 (+ 19us) server_negotiation.cc:407] Connection header received
0926 21:42:32.470569 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:32.470575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:32.470820 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:32.471006 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:32.471575 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.472592 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.474661 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:32.475248 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:32.475373 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:32.476057 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:32.476474 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:32.476598 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:32.476699 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 21:42:32.476784 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4900,"thread_start_us":4790,"threads_started":1}
I20260926 21:42:32.484357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22764
I20260926 21:42:32.485707 22826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:32.595614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 22764
I20260926 21:42:32.616602 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22838
I20260926 21:42:32.617748 22900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:32.725868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 22838
I20260926 21:42:32.746196 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 22912
I20260926 21:42:32.747318 22974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:32.854173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 22912
2026-09-26T21:42:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4475 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 21:42:32 dist-test-slave-lhhq krb5kdc[23162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 21:42:32 dist-test-slave-lhhq krb5kdc[23162](info): set up 2 sockets
Sep 26 21:42:32 dist-test-slave-lhhq krb5kdc[23162](info): commencing operation
krb5kdc: starting...
W20260926 21:42:33.985700 388 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/test-user.keytab.
Sep 26 21:42:34 dist-test-slave-lhhq krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T21:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T21:42:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 21:42:34.248385 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37609
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37609,127.0.97.61:40233,127.0.97.60:34047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 21:42:34.579682 23175 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:42:34.580085 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:34.580217 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:34.580365 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:34.587462 23175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:34.587617 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:34.587723 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:34.587811 23175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:34.587899 23175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:34.597750 23175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37609,127.0.97.61:40233,127.0.97.60:34047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:34.599114 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:34.600456 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:34.612622 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:34.612666 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:34.612579 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:34.614126 23175 server_base.cc:1034] running on GCE node
I20260926 21:42:34.614902 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:34.616295 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:34.617515 23175 hybrid_clock.cc:648] HybridClock initialized: now 1790458954617455 us; error 31 us; skew 500 ppm
I20260926 21:42:34.617825 23175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:34 dist-test-slave-lhhq krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:34.620966 23175 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 21:42:34.622572 23175 webserver.cc:460] Webserver started at http://127.0.97.62:38997/ using document root <none> and password file <none>
I20260926 21:42:34.623317 23175 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:34.623433 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:34.623747 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:34.626789 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/data/instance:
uuid: "739a17bcb2fc45748c19fcbe83ac963a"
format_stamp: "Formatted at 2026-09-26 21:42:34 on dist-test-slave-lhhq"
server_key: "ec1bacd3a7b69a299793c1049557b085"
server_key_iv: "9978edbe88995982a5f00a63fcde7a05"
server_key_version: "encryptionkey@0"
I20260926 21:42:34.627589 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/wal/instance:
uuid: "739a17bcb2fc45748c19fcbe83ac963a"
format_stamp: "Formatted at 2026-09-26 21:42:34 on dist-test-slave-lhhq"
server_key: "ec1bacd3a7b69a299793c1049557b085"
server_key_iv: "9978edbe88995982a5f00a63fcde7a05"
server_key_version: "encryptionkey@0"
I20260926 21:42:34.632622 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 21:42:34.635900 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:34.637156 23175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 21:42:34.637405 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/wal
uuid: "739a17bcb2fc45748c19fcbe83ac963a"
format_stamp: "Formatted at 2026-09-26 21:42:34 on dist-test-slave-lhhq"
server_key: "ec1bacd3a7b69a299793c1049557b085"
server_key_iv: "9978edbe88995982a5f00a63fcde7a05"
server_key_version: "encryptionkey@0"
I20260926 21:42:34.637611 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:34.650713 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:34.654618 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:34.654884 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:34.673454 23175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37609
I20260926 21:42:34.673482 23241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37609 every 8 connection(s)
I20260926 21:42:34.674793 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/data/info.pb
I20260926 21:42:34.678745 23242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:34.683573 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 23175
I20260926 21:42:34.683775 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-0/wal/instance
I20260926 21:42:34.684206 388 external_mini_cluster.cc:1428] Setting key c63186f98d9cb003bdb9eb2ebf7d9aaf
I20260926 21:42:34.686082 23242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:34 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:42:34.710497 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.689072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37609 (local address 127.0.0.1:55366)
0926 21:42:34.690089 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:34.690180 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:34.692022 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:34.692898 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:34.692910 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:34.693782 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:34.694272 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:34.694299 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:34.695833 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:34.695842 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:34.697745 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:34.697761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:34.698652 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:34.698660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:34.698872 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:34.699571 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:34.699604 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:34.702222 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:34.706565 (+ 4343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:34.706581 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:34.706590 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:34.706951 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:34.707445 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:34.707448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:34.707450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:34.707598 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:34.708624 (+ 1026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:34.708631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:34.708816 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 21:42:34.709016 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 21:42:34.709469 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":389,"threads_started":1}
I20260926 21:42:34.710551 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.689071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.0.97.62:37609)
0926 21:42:34.690739 (+ 1668us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:34.690749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:34.690804 (+ 55us) server_negotiation.cc:407] Connection header received
0926 21:42:34.692423 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:34.692429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:34.692653 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:34.692814 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:34.694484 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:34.695684 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:34.697878 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:34.698405 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:34.698529 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:34.702466 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:34.702478 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:34.702497 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:34.702538 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:34.706424 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:34.707097 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:34.707109 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:34.707116 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:34.707329 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:34.707718 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:34.707726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:34.707732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:34.708485 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:34.708707 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:34.709267 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 21:42:34.709658 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":1466,"spinlock_wait_cycles":12672,"thread_start_us":1281,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 21:42:34.713352 23242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:42:34.714565 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.714182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41166)
0926 21:42:34.714356 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:34.714453 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 21:42:34.714828 23194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
I20260926 21:42:34.716308 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.715972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41174)
0926 21:42:34.716126 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:34.716187 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 21:42:34.716490 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 21:42:34.743721 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40233
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37609,127.0.97.61:40233,127.0.97.60:34047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:42:34.765815 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 1
I20260926 21:42:34.766994 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.766517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41182)
0926 21:42:34.766752 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:34.766849 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 21:42:34.768029 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.767686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41188)
0926 21:42:34.767838 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:34.767900 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 21:42:34.768220 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
I20260926 21:42:34.836524 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 2
I20260926 21:42:34.837822 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.837315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41204)
0926 21:42:34.837564 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:34.837651 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 21:42:34.838883 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.838510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41216)
0926 21:42:34.838687 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:34.838751 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 21:42:34.839053 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
I20260926 21:42:34.930306 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 3
I20260926 21:42:34.931663 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.930993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41218)
0926 21:42:34.931283 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:34.931452 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 21:42:34.932945 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:34.932431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41220)
0926 21:42:34.932629 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:34.932730 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 21:42:34.933173 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
W20260926 21:42:35.066510 23249 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:42:35.066854 23249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:35.066919 23249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:35.067003 23249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:35.074200 23249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:35.074303 23249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:35.074360 23249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:35.074404 23249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:35.074467 23249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:35.076500 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 4
I20260926 21:42:35.077795 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.077157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41230)
0926 21:42:35.077509 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.077630 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260926 21:42:35.078850 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.078463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41238)
0926 21:42:35.078617 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.078703 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 21:42:35.079030 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
I20260926 21:42:35.085366 23249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37609,127.0.97.61:40233,127.0.97.60:34047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:35.086624 23249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:35.087858 23249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:35.099335 23254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:35.099350 23255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:35.099555 23257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:35.100035 23249 server_base.cc:1034] running on GCE node
I20260926 21:42:35.100781 23249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:35.102192 23249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:35.103440 23249 hybrid_clock.cc:648] HybridClock initialized: now 1790458955103384 us; error 53 us; skew 500 ppm
I20260926 21:42:35.103772 23249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:35.107031 23249 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 21:42:35.108444 23249 webserver.cc:460] Webserver started at http://127.0.97.61:36313/ using document root <none> and password file <none>
I20260926 21:42:35.109266 23249 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:35.109418 23249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:35.109740 23249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:35.112625 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/data/instance:
uuid: "1285c54868064696b4797f6526ae00ca"
format_stamp: "Formatted at 2026-09-26 21:42:35 on dist-test-slave-lhhq"
server_key: "c52b246088ffb22efb2721179106ffac"
server_key_iv: "12e92d16e741ec4592cb68c1df507998"
server_key_version: "encryptionkey@0"
I20260926 21:42:35.113534 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/wal/instance:
uuid: "1285c54868064696b4797f6526ae00ca"
format_stamp: "Formatted at 2026-09-26 21:42:35 on dist-test-slave-lhhq"
server_key: "c52b246088ffb22efb2721179106ffac"
server_key_iv: "12e92d16e741ec4592cb68c1df507998"
server_key_version: "encryptionkey@0"
I20260926 21:42:35.118553 23249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 21:42:35.121912 23264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:35.123160 23249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:42:35.123358 23249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/wal
uuid: "1285c54868064696b4797f6526ae00ca"
format_stamp: "Formatted at 2026-09-26 21:42:35 on dist-test-slave-lhhq"
server_key: "c52b246088ffb22efb2721179106ffac"
server_key_iv: "12e92d16e741ec4592cb68c1df507998"
server_key_version: "encryptionkey@0"
I20260926 21:42:35.123575 23249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:35.144961 23249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:35.148917 23249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:35.149233 23249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:35.167567 23249 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40233
I20260926 21:42:35.167585 23315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40233 every 8 connection(s)
I20260926 21:42:35.168949 23249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/data/info.pb
I20260926 21:42:35.172894 23316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:35.177646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 23249
I20260926 21:42:35.177840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-1/wal/instance
I20260926 21:42:35.178269 388 external_mini_cluster.cc:1428] Setting key ef010e4aa2d59804d10d0b3dbb2cd586
I20260926 21:42:35.179339 23316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:42:35.201179 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.182037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.0.97.62:37609)
0926 21:42:35.182272 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.182281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.183396 (+ 1115us) server_negotiation.cc:407] Connection header received
0926 21:42:35.186379 (+ 2983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.186388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.186479 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.186573 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.188248 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.188932 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.190462 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.190892 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.191039 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.194903 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.194909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.194913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.194949 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.197762 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.198796 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.198801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.198804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.198937 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.199431 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.199434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.199436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.199704 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.199865 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.200617 (+ 752us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.200849 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:42:35.202261 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.182395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37609 (local address 127.0.0.1:55372)
0926 21:42:35.183186 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.183245 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:35.186052 (+ 2807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:35.186853 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:35.186868 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:35.187508 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:35.188062 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.188079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.189066 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.189070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.190312 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.190324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.191114 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.191120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.191355 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.192053 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:35.192075 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:35.194661 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:35.197917 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.197928 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.197948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.198645 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.199050 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.199058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.199064 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.199320 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.199859 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:35.199867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:35.200232 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 21:42:35.200504 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 21:42:35.201055 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":301,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260926 21:42:35.204275 23316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:42:35.220820 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.205053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41252)
0926 21:42:35.205298 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.205320 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:35.205435 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:35.206427 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:35.206431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:35.206918 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:35.207075 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.207084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.208473 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.208482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.210054 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.210071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.210923 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.210931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.211154 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.211774 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:35.211809 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:35.214262 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:35.218047 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.218053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.218058 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.218477 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.218882 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.218891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.218895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.219028 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.219637 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:35.219642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:35.219883 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 21:42:35.220030 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 21:42:35.220393 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 21:42:35.220947 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.205351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.0.97.61:40233)
0926 21:42:35.205946 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.205952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.205977 (+ 25us) server_negotiation.cc:407] Connection header received
0926 21:42:35.206019 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.206025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.206272 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.206441 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.207277 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.208314 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.210192 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.210748 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.210865 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.214473 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.214485 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.214498 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.214530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.217892 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.218628 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.218643 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.218645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.218769 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.219142 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.219153 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.219156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.219510 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.219770 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.220197 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.220519 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":455,"thread_start_us":323,"threads_started":1}
I20260926 21:42:35.223745 23316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:42:35.224901 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.224431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33174)
0926 21:42:35.224600 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.224732 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 21:42:35.225152 23265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
I20260926 21:42:35.227150 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.226530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33176)
0926 21:42:35.226691 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.227015 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 21:42:35.227370 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34047: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 21:42:35.239981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
/tmp/dist-test-taskh3AZIa/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34047
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37609,127.0.97.61:40233,127.0.97.60:34047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 21:42:35.276710 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } attempt: 1
I20260926 21:42:35.277930 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.277468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33192)
0926 21:42:35.277673 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.277768 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 21:42:35.278904 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.278568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33194)
0926 21:42:35.278707 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.278798 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 21:42:35.279132 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34047: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
I20260926 21:42:35.347378 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } attempt: 2
I20260926 21:42:35.348553 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.348087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33204)
0926 21:42:35.348293 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.348394 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 21:42:35.349651 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.349326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33206)
0926 21:42:35.349468 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.349542 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 21:42:35.349908 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34047: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
I20260926 21:42:35.378283 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 5
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:42:35.393473 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.379180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.0.97.61:40233)
0926 21:42:35.379359 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.379363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.379417 (+ 54us) server_negotiation.cc:407] Connection header received
0926 21:42:35.379605 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.379609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.379720 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.379850 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.380674 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.381457 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.382620 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.383032 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.383167 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.386782 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.386788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.386792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.386831 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.390312 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.391114 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.391122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.391129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.391284 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.391757 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.391766 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.391769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.392121 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.392310 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.392767 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.393031 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 21:42:35.393478 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.379077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41268)
0926 21:42:35.379313 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.379335 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:35.379484 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:35.379880 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:35.379885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:35.380328 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:35.380507 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.380516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.381611 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.381616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.382471 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.382482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.383190 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.383195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.383385 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.384042 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:35.384060 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:35.386458 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:35.390500 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.390507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.390512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.390963 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.391446 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.391452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.391457 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.391627 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.392265 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:35.392271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:35.392460 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 21:42:35.392626 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 21:42:35.392974 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 21:42:35.394732 23242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:42:35.395856 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.395437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33214)
0926 21:42:35.395611 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.395697 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 21:42:35.396908 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.396587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33228)
0926 21:42:35.396723 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.396789 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 21:42:35.397241 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34047: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
I20260926 21:42:35.441087 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } attempt: 3
I20260926 21:42:35.442440 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.441919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33236)
0926 21:42:35.442126 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.442247 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 21:42:35.443454 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.443051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33240)
0926 21:42:35.443263 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.443353 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260926 21:42:35.443645 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34047: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
I20260926 21:42:35.448478 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } attempt: 1
I20260926 21:42:35.449555 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.449088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33254)
0926 21:42:35.449318 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.449400 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 21:42:35.450539 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.450179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33268)
0926 21:42:35.450364 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.450436 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 21:42:35.450739 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34047: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
I20260926 21:42:35.518947 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } attempt: 2
I20260926 21:42:35.520109 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.519632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33270)
0926 21:42:35.519830 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.519940 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 21:42:35.521023 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.520718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33278)
0926 21:42:35.520855 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.520927 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 21:42:35.521241 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34047: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
W20260926 21:42:35.578241 23323 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 21:42:35.578665 23323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 21:42:35.578805 23323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 21:42:35.578950 23323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 21:42:35.586512 23323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 21:42:35.586658 23323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 21:42:35.586767 23323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 21:42:35.586858 23323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 21:42:35.586970 23323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 21:42:35.586876 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } attempt: 4
I20260926 21:42:35.588032 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.587556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33290)
0926 21:42:35.587782 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.587881 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 21:42:35.589145 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.588800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33306)
0926 21:42:35.588973 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.589043 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 21:42:35.589385 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34047: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
I20260926 21:42:35.597683 23323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37609,127.0.97.61:40233,127.0.97.60:34047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 20:26:36 UTC on ccycloud.quasar-bgclim.root.comops.site
build id 4675384
ASAN enabled
I20260926 21:42:35.599069 23323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 21:42:35.600390 23323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 21:42:35.612938 23328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:35.612922 23329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 21:42:35.612887 23331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 21:42:35.614152 23323 server_base.cc:1034] running on GCE node
I20260926 21:42:35.615046 23323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 21:42:35.616614 23323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 21:42:35.617861 23323 hybrid_clock.cc:648] HybridClock initialized: now 1790458955617777 us; error 63 us; skew 500 ppm
I20260926 21:42:35.618237 23323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 21:42:35.621663 23323 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 21:42:35.623171 23323 webserver.cc:460] Webserver started at http://127.0.97.60:34151/ using document root <none> and password file <none>
I20260926 21:42:35.623904 23323 fs_manager.cc:359] Metadata directory not provided
I20260926 21:42:35.623992 23323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 21:42:35.624266 23323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 21:42:35.627429 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } attempt: 3
I20260926 21:42:35.627420 23323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/data/instance:
uuid: "87c7910f1ec84a8386dbbb673871cdc3"
format_stamp: "Formatted at 2026-09-26 21:42:35 on dist-test-slave-lhhq"
server_key: "aac8d5e59f6821559012b4c78d11089c"
server_key_iv: "b0faefbd8af51009a1a17cab49d477dc"
server_key_version: "encryptionkey@0"
I20260926 21:42:35.628315 23323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/wal/instance:
uuid: "87c7910f1ec84a8386dbbb673871cdc3"
format_stamp: "Formatted at 2026-09-26 21:42:35 on dist-test-slave-lhhq"
server_key: "aac8d5e59f6821559012b4c78d11089c"
server_key_iv: "b0faefbd8af51009a1a17cab49d477dc"
server_key_version: "encryptionkey@0"
I20260926 21:42:35.628590 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.628077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33322)
0926 21:42:35.628303 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.628384 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 21:42:35.629637 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.629253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33326)
0926 21:42:35.629461 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.629541 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 21:42:35.629859 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34047: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34047: connect: Connection refused (error 111)
I20260926 21:42:35.633729 23323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 21:42:35.637138 23338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 21:42:35.638502 23323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 21:42:35.638687 23323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/wal
uuid: "87c7910f1ec84a8386dbbb673871cdc3"
format_stamp: "Formatted at 2026-09-26 21:42:35 on dist-test-slave-lhhq"
server_key: "aac8d5e59f6821559012b4c78d11089c"
server_key_iv: "b0faefbd8af51009a1a17cab49d477dc"
server_key_version: "encryptionkey@0"
I20260926 21:42:35.638913 23323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 21:42:35.666013 23323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 21:42:35.670280 23323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 21:42:35.670601 23323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 21:42:35.690091 23323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34047
I20260926 21:42:35.690099 23389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34047 every 8 connection(s)
I20260926 21:42:35.691524 23323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/data/info.pb
I20260926 21:42:35.695457 23390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 21:42:35.695969 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu as pid 23323
I20260926 21:42:35.696151 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh3AZIa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790458593267894-388-0/minicluster-data/master-2/wal/instance
I20260926 21:42:35.696709 388 external_mini_cluster.cc:1428] Setting key 80e2ffcfb5420b7fba389eeda73b22b6
I20260926 21:42:35.704996 23390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:42:35.721735 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.698514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.0.97.60:34047)
0926 21:42:35.699321 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.699332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.699373 (+ 41us) server_negotiation.cc:407] Connection header received
0926 21:42:35.701723 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.701746 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.702190 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.702537 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.704253 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.705599 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.707366 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.708503 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.708742 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.713106 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.713117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.713132 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.713222 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.716785 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.717632 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.717643 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.717653 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.717830 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.718235 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.718246 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.718248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.718954 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.719264 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.719741 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.720172 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":400,"threads_started":1}
I20260926 21:42:35.724262 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.707915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37609 (local address 127.0.0.1:55388)
0926 21:42:35.708994 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.709042 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:35.709231 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:35.709695 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:35.709700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:35.710285 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:35.710522 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.710530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.711901 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.711906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.713009 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.713021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.713776 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.713780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.713907 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.714493 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:35.714516 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:35.717124 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:35.721009 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.721024 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.721030 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.721779 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.722281 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.722286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.722289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.722477 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.723192 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:35.723200 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:35.723405 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 21:42:35.723587 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 21:42:35.723878 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":584,"threads_started":1}
I20260926 21:42:35.724661 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.707891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.0.97.62:37609)
0926 21:42:35.708570 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.708576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.709160 (+ 584us) server_negotiation.cc:407] Connection header received
0926 21:42:35.709396 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.709400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.709511 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.709626 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.710677 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.711751 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.713149 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.713598 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.713690 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.717338 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.717344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.717348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.717380 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.720836 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.721970 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.721975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.721978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.722135 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.722620 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.722624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.722659 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.723073 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.723208 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.723748 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.723973 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":371,"threads_started":1}
I20260926 21:42:35.732287 23390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 21:42:35.748018 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.733029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:41278)
0926 21:42:35.733269 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.733288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:35.733380 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:35.734130 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:35.734136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:35.734610 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:35.734822 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.734832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.736371 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.736376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.737784 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.737815 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.738721 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.738729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.738903 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.739370 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:35.739396 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:35.742097 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:35.745356 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.745362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.745366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.745883 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.746247 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.746255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.746261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.746439 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.747075 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:35.747078 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:35.747229 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 21:42:35.747381 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 21:42:35.747674 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 21:42:35.748055 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.733589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.0.97.61:40233)
0926 21:42:35.733795 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.733810 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.733828 (+ 18us) server_negotiation.cc:407] Connection header received
0926 21:42:35.733890 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.733898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.733973 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.734087 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.735259 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.736231 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.737929 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.738563 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.738651 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.742371 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.742381 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.742384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.742416 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.745170 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.746002 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.746006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.746008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.746140 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.746593 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.746597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.746600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.746955 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.747115 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.747476 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.747730 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:42:35.749118 23390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } has no permanent_uuid. Determining permanent_uuid...
I20260926 21:42:35.749825 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.732431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.0.97.62:37609)
0926 21:42:35.732603 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.732609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.732747 (+ 138us) server_negotiation.cc:407] Connection header received
0926 21:42:35.732964 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.732970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.733079 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.734730 (+ 1651us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.734862 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.735974 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.737694 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.738224 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.738357 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.743128 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.743136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.743141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.743177 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.746471 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.747276 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.747281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.747285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.747445 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.747856 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.747861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.747865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.748453 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.748759 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.749039 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.749331 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 21:42:35.764135 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.749837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33336)
0926 21:42:35.750027 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.750046 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:35.750150 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:35.750533 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:35.750537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:35.751000 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:35.751173 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.751184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.752510 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.752513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.753545 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.753558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.754229 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.754238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.754422 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.754990 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:35.755006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:35.757781 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:35.761483 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.761491 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.761496 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.761874 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.762301 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.762308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.762312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.762506 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.763261 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:35.763264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:35.763463 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 21:42:35.763637 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 21:42:35.763856 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 21:42:35.764497 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.750094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.0.97.60:34047)
0926 21:42:35.750240 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.750246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.750263 (+ 17us) server_negotiation.cc:407] Connection header received
0926 21:42:35.750312 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.750323 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.750394 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.750496 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.751326 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.752386 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.753673 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.754063 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.754161 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.758002 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.758012 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.758018 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.758067 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.761331 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.762000 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.762006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.762011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.762160 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.762621 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.762626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.762675 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.763104 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.763283 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.763779 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.764056 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 21:42:35.767882 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.751427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.0.97.61:40233)
0926 21:42:35.751623 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.751627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.751656 (+ 29us) server_negotiation.cc:407] Connection header received
0926 21:42:35.751799 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.751802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.751853 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.751921 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.753251 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.754103 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.755312 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.756074 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.756232 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.761588 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.761593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.761597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.761631 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.765090 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.765799 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.765805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.765808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.765990 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.766408 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.766413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.766419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.766778 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.767051 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.767317 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.767553 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 21:42:35.770699 23390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3: Bootstrap starting.
I20260926 21:42:35.773151 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 21:42:35.774631 23390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:35.775936 23390 log.cc:824] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:35.778824 23390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3: No bootstrap required, opened a new log
I20260926 21:42:35.776367 23217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "739a17bcb2fc45748c19fcbe83ac963a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55394
I20260926 21:42:35.785925 23390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } }
I20260926 21:42:35.786273 23390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c7910f1ec84a8386dbbb673871cdc3, State: Initialized, Role: FOLLOWER
I20260926 21:42:35.786705 23390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } }
I20260926 21:42:35.787801 23397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:35.788265 23397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:35.788707 23390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 21:42:35.798386 23408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:42:35.798547 23408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 21:42:35.807061 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } attempt: 4
I20260926 21:42:35.796233 23323 master_runner.cc:186] Error getting master registration for 127.0.97.62:37609: OK, instance_id { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" instance_seqno: 1790458954668022 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:42:35.807706 23323 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:42:35.824424 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.807821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33338)
0926 21:42:35.808031 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.808054 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:35.808189 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:35.808724 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:35.808729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:35.809267 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:35.809494 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.809508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.811069 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.811074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.812553 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.812570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.813688 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.813695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.813920 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.814698 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:35.814724 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:35.817640 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:35.821793 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.821799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.821804 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.822203 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.822615 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.822620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.822624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.822789 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.823381 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:35.823386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:35.823540 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 21:42:35.823700 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 21:42:35.824028 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 21:42:35.824420 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.808099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.0.97.60:34047)
0926 21:42:35.808336 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.808344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.808365 (+ 21us) server_negotiation.cc:407] Connection header received
0926 21:42:35.808440 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.808445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.808541 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.808661 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.809666 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.810901 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.812722 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.813523 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.813648 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.817861 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.817873 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.817876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.817912 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.821633 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.822342 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.822347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.822351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.822488 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.822900 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.822906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.822910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.823223 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.823395 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.823819 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.824036 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 21:42:35.830471 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a: Bootstrap starting.
I20260926 21:42:35.834643 23242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:35.835991 23242 log.cc:824] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:35.840744 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a: No bootstrap required, opened a new log
I20260926 21:42:35.848549 23242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } }
I20260926 21:42:35.849009 23242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739a17bcb2fc45748c19fcbe83ac963a, State: Initialized, Role: FOLLOWER
I20260926 21:42:35.849567 23242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } }
I20260926 21:42:35.850632 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:35.851023 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:35.851923 23242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:35.851966 23217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 21:42:35.852185 23217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:35.854992 23217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } }
I20260926 21:42:35.857209 23217 leader_election.cc:288] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [CANDIDATE]: Term 1 election: Requested vote from peers 1285c54868064696b4797f6526ae00ca (127.0.97.61:40233), 87c7910f1ec84a8386dbbb673871cdc3 (127.0.97.60:34047)
I20260926 21:42:35.857161 23365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "739a17bcb2fc45748c19fcbe83ac963a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "87c7910f1ec84a8386dbbb673871cdc3"
I20260926 21:42:35.857754 23365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:35.861991 23365 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739a17bcb2fc45748c19fcbe83ac963a in term 1.
I20260926 21:42:35.862699 23191 leader_election.cc:302] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 739a17bcb2fc45748c19fcbe83ac963a, 87c7910f1ec84a8386dbbb673871cdc3; no voters:
I20260926 21:42:35.863235 23411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 21:42:35.873500 23411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [term 1 LEADER]: Becoming Leader. State: Replica: 739a17bcb2fc45748c19fcbe83ac963a, State: Running, Role: LEADER
I20260926 21:42:35.874276 23411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } }
I20260926 21:42:35.887635 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 739a17bcb2fc45748c19fcbe83ac963a. Latest consensus state: current_term: 1 leader_uuid: "739a17bcb2fc45748c19fcbe83ac963a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:35.888119 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:35.888583 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } attempt: 5
W20260926 21:42:35.909485 23196 tablet.cc:2369] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 21:42:35.921305 23432 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 21:42:35.921741 23432 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 21:42:35.922852 23344 tablet.cc:2369] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 21:42:35.925050 23431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 21:42:35.931645 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.883737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.0.97.62:37609)
0926 21:42:35.884232 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.884240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.887978 (+ 3738us) server_negotiation.cc:407] Connection header received
0926 21:42:35.897652 (+ 9674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.897660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.897776 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.897865 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.903790 (+ 5925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.905430 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.912265 (+ 6835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.913410 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.913576 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.924225 (+ 10649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.924235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.924241 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.924293 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.928150 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.928953 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.928959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.928963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.929138 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.929648 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.929653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.929657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.930075 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.930375 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.930919 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.931223 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345}
I20260926 21:42:35.932253 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.883994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.0.97.61:40233)
0926 21:42:35.884382 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.884387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.897510 (+ 13123us) server_negotiation.cc:407] Connection header received
0926 21:42:35.900925 (+ 3415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.900933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.901023 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.901099 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.907972 (+ 6873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.909503 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.914963 (+ 5460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.916096 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.916238 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.923289 (+ 7051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.923298 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.923302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.923345 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.928129 (+ 4784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.929648 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.929654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.929673 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.929819 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.930270 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.930275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.930280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.930644 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.930869 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.931410 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.931635 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260926 21:42:35.942610 23431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 21:42:35 dist-test-slave-lhhq krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790458955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 21:42:35.905829 23175 master_runner.cc:186] Error getting master registration for 127.0.97.61:40233: OK, instance_id { permanent_uuid: "1285c54868064696b4797f6526ae00ca" instance_seqno: 1790458955162322 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 21:42:35.946336 23175 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 21:42:35.962762 23365 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [term 1 FOLLOWER]: Refusing update from remote peer 739a17bcb2fc45748c19fcbe83ac963a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:42:35.963788 23411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:42:35.970650 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.889176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34047 (local address 127.0.0.1:33350)
0926 21:42:35.889581 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 21:42:35.889613 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 21:42:35.889740 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 21:42:35.899635 (+ 9895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 21:42:35.899641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 21:42:35.900142 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 21:42:35.900358 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.900371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.927232 (+ 26861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.927239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.928875 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 21:42:35.928891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.940030 (+ 11139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.940037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 21:42:35.940249 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.940957 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 21:42:35.940984 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 21:42:35.944865 (+ 3881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 21:42:35.964564 (+ 19699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.964571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.964575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.965160 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.967539 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 21:42:35.967545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 21:42:35.967549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 21:42:35.967722 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 21:42:35.969556 (+ 1834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 21:42:35.969561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 21:42:35.969735 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 21:42:35.969937 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 21:42:35.970241 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260926 21:42:35.972215 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.888379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.0.97.60:34047)
0926 21:42:35.888558 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.888568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.912914 (+ 24346us) server_negotiation.cc:407] Connection header received
0926 21:42:35.913641 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.913649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.913732 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.913813 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.935487 (+ 21674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.936601 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.947609 (+ 11008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.948597 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.948732 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.956247 (+ 7515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.956254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.956260 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.956303 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.960562 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.961495 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.961502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.961508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.962319 (+ 811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.963267 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.963273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.963276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.965719 (+ 2443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.968115 (+ 2396us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.968497 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.969384 (+ 887us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 21:42:35.973917 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:35.890211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.0.97.60:34047)
0926 21:42:35.897969 (+ 7758us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:35.897976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:35.898100 (+ 124us) server_negotiation.cc:407] Connection header received
0926 21:42:35.898370 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:35.898376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:35.898717 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:35.900693 (+ 1976us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:35.913096 (+ 12403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.926467 (+ 13371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.929847 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:35.939288 (+ 9441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:35.945137 (+ 5849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:35.946102 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 21:42:35.946112 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 21:42:35.946210 (+ 98us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 21:42:35.946351 (+ 141us) server_negotiation.cc:922] calling sasl_server_start()
0926 21:42:35.964236 (+ 17885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.965649 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.965657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.965661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.967220 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 21:42:35.968054 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 21:42:35.968060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 21:42:35.968069 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 21:42:35.969270 (+ 1201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 21:42:35.971130 (+ 1860us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:35.971644 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 21:42:35.973056 (+ 1412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7223,"thread_start_us":6390,"threads_started":1}
I20260926 21:42:35.980183 23316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca: Bootstrap starting.
I20260926 21:42:35.984334 23316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca: Neither blocks nor log segments found. Creating new log.
I20260926 21:42:35.986333 23316 log.cc:824] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca: Log is configured to *not* fsync() on all Append() calls
I20260926 21:42:35.989416 23397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 739a17bcb2fc45748c19fcbe83ac963a. Latest consensus state: current_term: 1 leader_uuid: "739a17bcb2fc45748c19fcbe83ac963a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:35.989799 23397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:35.997283 23435 mvcc.cc:205] Tried to move back new op lower bound from 7333719883608653824 to 7333719883268587520. Current Snapshot: MvccSnapshot[applied={T|T < 7333719883608653824}]
I20260926 21:42:36.000527 23436 mvcc.cc:205] Tried to move back new op lower bound from 7333719883608653824 to 7333719883268587520. Current Snapshot: MvccSnapshot[applied={T|T < 7333719883608653824}]
I20260926 21:42:36.001575 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "739a17bcb2fc45748c19fcbe83ac963a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:36.001879 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:36.008388 23431 catalog_manager.cc:1347] Generated new cluster ID: 70b7fcf4780d41689dba4414badc2724
I20260926 21:42:36.008508 23431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 21:42:36.012167 23397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "739a17bcb2fc45748c19fcbe83ac963a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:36.012456 23397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:36.018319 23316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca: No bootstrap required, opened a new log
I20260926 21:42:36.028614 23431 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 21:42:36.030619 23431 catalog_manager.cc:1504] Loading token signing keys...
I20260926 21:42:36.040623 23316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } }
I20260926 21:42:36.041289 23316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1285c54868064696b4797f6526ae00ca, State: Initialized, Role: FOLLOWER
I20260926 21:42:36.042173 23316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } }
I20260926 21:42:36.058604 23441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:36.059224 23441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:36.060485 23431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a: Generated new TSK 0
I20260926 21:42:36.061619 23431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 21:42:36.062728 23291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "739a17bcb2fc45748c19fcbe83ac963a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1285c54868064696b4797f6526ae00ca"
I20260926 21:42:36.064556 23290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 21:42:36.065749 23316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 21:42:36.068374 23290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [term 1 FOLLOWER]: Refusing update from remote peer 739a17bcb2fc45748c19fcbe83ac963a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 21:42:36.071962 23291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739a17bcb2fc45748c19fcbe83ac963a in term 1.
I20260926 21:42:36.074839 23411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 21:42:36.119370 23441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 739a17bcb2fc45748c19fcbe83ac963a. Latest consensus state: current_term: 1 leader_uuid: "739a17bcb2fc45748c19fcbe83ac963a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:36.119724 23441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:36.120093 23446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "739a17bcb2fc45748c19fcbe83ac963a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:36.120604 23446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 21:42:36.201069 23463 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 21:42:36.213275 23394 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.197160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.0.97.62:37609)
0926 21:42:36.197382 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.197391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.211384 (+ 13993us) server_negotiation.cc:407] Connection header received
0926 21:42:36.211683 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.211688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.211776 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.211850 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.213013 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55404: BlockingRecv error: recv got EOF from 127.0.0.1:55404 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 21:42:36.215454 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.208492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.0.97.61:40233)
0926 21:42:36.208689 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.208695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.213974 (+ 5279us) server_negotiation.cc:407] Connection header received
0926 21:42:36.214185 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.214191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.214281 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.214346 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.215235 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41308: BlockingRecv error: recv got EOF from 127.0.0.1:41308 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 21:42:36.218787 23433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.216822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.0.97.60:34047)
0926 21:42:36.216985 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.216990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.217257 (+ 267us) server_negotiation.cc:407] Connection header received
0926 21:42:36.217422 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.217429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.217520 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.217635 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.218556 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33366: BlockingRecv error: recv got EOF from 127.0.0.1:33366 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 21:42:36.231212 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.222452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.0.97.61:40233)
0926 21:42:36.222625 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.222631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.229707 (+ 7076us) server_negotiation.cc:407] Connection header received
0926 21:42:36.229929 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.229937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.230071 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.230183 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.230969 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41316: BlockingRecv error: recv got EOF from 127.0.0.1:41316 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 21:42:36.242877 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.235928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.97.61:40233)
0926 21:42:36.236087 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.236093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.241180 (+ 5087us) server_negotiation.cc:407] Connection header received
0926 21:42:36.241469 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.241477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.241577 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.241691 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.242645 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41320: BlockingRecv error: recv got EOF from 127.0.0.1:41320 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 21:42:36.254398 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.252065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41328 (local address 127.0.97.61:40233)
0926 21:42:36.252247 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.252253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.252772 (+ 519us) server_negotiation.cc:407] Connection header received
0926 21:42:36.252978 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.252985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.253109 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.253252 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.254154 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41328: BlockingRecv error: recv got EOF from 127.0.0.1:41328 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 21:42:36.273746 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.271353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.0.97.61:40233)
0926 21:42:36.271586 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.271593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.272138 (+ 545us) server_negotiation.cc:407] Connection header received
0926 21:42:36.272350 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.272356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.272457 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.272579 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.273498 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41330: BlockingRecv error: recv got EOF from 127.0.0.1:41330 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260926 21:42:36.287228 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.285038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.0.97.61:40233)
0926 21:42:36.285262 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.285267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.285751 (+ 484us) server_negotiation.cc:407] Connection header received
0926 21:42:36.285938 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.285944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.286042 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.286165 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.286995 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41344: BlockingRecv error: recv got EOF from 127.0.0.1:41344 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 21:42:36.308732 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.306561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.0.97.61:40233)
0926 21:42:36.306756 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.306763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.307277 (+ 514us) server_negotiation.cc:407] Connection header received
0926 21:42:36.307471 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.307477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.307573 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.307700 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.308491 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41352: BlockingRecv error: recv got EOF from 127.0.0.1:41352 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 21:42:36.349304 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.346896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.0.97.61:40233)
0926 21:42:36.347089 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.347096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.347622 (+ 526us) server_negotiation.cc:407] Connection header received
0926 21:42:36.347862 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.347868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.347968 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.348083 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.349008 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41356: BlockingRecv error: recv got EOF from 127.0.0.1:41356 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 21:42:36.418385 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.416422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.0.97.61:40233)
0926 21:42:36.416678 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.416686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.417040 (+ 354us) server_negotiation.cc:407] Connection header received
0926 21:42:36.417240 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.417247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.417355 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.417484 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.418162 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41358: BlockingRecv error: recv got EOF from 127.0.0.1:41358 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 21:42:36.553946 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.552005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.0.97.61:40233)
0926 21:42:36.552267 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.552275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.552555 (+ 280us) server_negotiation.cc:407] Connection header received
0926 21:42:36.552708 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.552712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.552827 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.552917 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.553700 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41364: BlockingRecv error: recv got EOF from 127.0.0.1:41364 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 21:42:36.599188 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "739a17bcb2fc45748c19fcbe83ac963a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a17bcb2fc45748c19fcbe83ac963a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37609 } } peers { permanent_uuid: "1285c54868064696b4797f6526ae00ca" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "87c7910f1ec84a8386dbbb673871cdc3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34047 } } }
I20260926 21:42:36.599545 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739a17bcb2fc45748c19fcbe83ac963a [sys.catalog]: This master's current role is: LEADER
I20260926 21:42:36.800554 23408 catalog_manager.cc:1259] Loaded cluster ID: 70b7fcf4780d41689dba4414badc2724
I20260926 21:42:36.800665 23408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3: loading cluster ID for follower catalog manager: success
I20260926 21:42:36.803218 23408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3: acquiring CA information for follower catalog manager: success
I20260926 21:42:36.805100 23408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87c7910f1ec84a8386dbbb673871cdc3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 21:42:36.816690 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 21:42:36.814797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.0.97.61:40233)
0926 21:42:36.815015 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:36.815019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:36.815325 (+ 306us) server_negotiation.cc:407] Connection header received
0926 21:42:36.815483 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:36.815487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:36.815566 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:36.815669 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 21:42:36.816514 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41374: BlockingRecv error: recv got EOF from 127.0.0.1:41374 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 21:42:37.205415 23463 catalog_manager.cc:1259] Loaded cluster ID: 70b7fcf4780d41689dba4414badc2724
I20260926 21:42:37.205534 23463 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca: loading cluster ID for follower catalog manager: success
I20260926 21:42:37.208098 23463 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca: acquiring CA information for follower catalog manager: success
I20260926 21:42:37.210162 23463 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1285c54868064696b4797f6526ae00ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 21:42:37.349664 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:37.342444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.0.97.61:40233)
0926 21:42:37.343180 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:37.343186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:37.343438 (+ 252us) server_negotiation.cc:407] Connection header received
0926 21:42:37.343653 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:37.343657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:37.343828 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:37.343942 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:37.344415 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.345309 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.347312 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.347737 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.347844 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:37.348396 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:37.348971 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:37.349078 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:37.349264 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 21:42:37.349331 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":360,"threads_started":1}
I20260926 21:42:37.363090 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:37.355915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.0.97.60:34047)
0926 21:42:37.356432 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:37.356439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:37.356753 (+ 314us) server_negotiation.cc:407] Connection header received
0926 21:42:37.356963 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:37.356969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:37.357175 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:37.357391 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:37.357734 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.358857 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.360276 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.360920 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.361103 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:37.361636 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:37.362426 (+ 790us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:37.362570 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:37.362663 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 21:42:37.362753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":270,"threads_started":1}
I20260926 21:42:37.377416 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:37.369707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.0.97.61:40233)
0926 21:42:37.370012 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:37.370019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:37.370259 (+ 240us) server_negotiation.cc:407] Connection header received
0926 21:42:37.370542 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:37.370551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:37.370678 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:37.370797 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:37.371281 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.372942 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.374933 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.375838 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.375972 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:37.376518 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:37.376649 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:37.376714 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:37.376966 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 21:42:37.377053 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260926 21:42:37.385113 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:37.377456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.0.97.60:34047)
0926 21:42:37.377638 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:37.377643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:37.378142 (+ 499us) server_negotiation.cc:407] Connection header received
0926 21:42:37.379043 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:37.379048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:37.379142 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:37.379211 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:37.379769 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.381096 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.383112 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.383746 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.383889 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:37.384275 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:37.384454 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:37.384552 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:37.384729 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 21:42:37.384815 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 21:42:37.389726 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 21:42:37.369703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.0.97.62:37609)
0926 21:42:37.383754 (+ 14051us) server_negotiation.cc:206] Beginning negotiation
0926 21:42:37.383760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 21:42:37.383780 (+ 20us) server_negotiation.cc:407] Connection header received
0926 21:42:37.383926 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 21:42:37.383932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 21:42:37.384139 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 21:42:37.384336 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 21:42:37.384968 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.385835 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.387642 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 21:42:37.388035 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 21:42:37.388178 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 21:42:37.388710 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 21:42:37.389080 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 21:42:37.389216 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 21:42:37.389337 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 21:42:37.389418 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13918,"thread_start_us":13814,"threads_started":1}
I20260926 21:42:37.403553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 23175
I20260926 21:42:37.404721 23237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:37.517580 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 23175
W20260926 21:42:37.538643 23265 connection.cc:535] server connection from 127.0.0.1:41268 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 21:42:37.539081 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 23249
I20260926 21:42:37.540174 23311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:37.653324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 23249
I20260926 21:42:37.673458 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu pid 23323
I20260926 21:42:37.674576 23385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 21:42:37.785454 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh3AZIa/build/asan/bin/kudu with pid 23323
2026-09-26T21:42:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4931 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9407 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (364480 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST